Using OpenAI's SDK, the following function
from openai import OpenAI
client = OpenAI(
api_key="My OPENAI Key", # defaults to os.environ.get("OPENAI_API_KEY")
)
chat_completion = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
)
becomes
from openai import OpenAI
client = OpenAI(
api_key="My MARTIAN Key", # defaults to os.environ.get("OPENAI_API_KEY")
base_url="https://withmartian.com/api/openai/v1",
)
chat_completion = client.chat.completions.create(
model="router",
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
)
Setting Parameters
To route across a subset of models, specifiy the array of whitelisted origins:
chat_completion = client.chat.completions.create(
model="router",
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
extra_body={
"models": ["gpt-3.5-turbo", "claude-2.1"]
}
)
To specify a max cost and willingness to pay for a 10% improvement on model quality on each request, set the parameters in the following way:\
client.chat.completions.create(
model="router",
messages=[
{"role": "user", "content": "Hello world!"}
],
extra_body={
"models": ["gpt-3.5-turbo", "claude-2.1"],
"max_cost": 0.02,
"willingness_to_pay": 0.01
},
)
Finally, to associate metadata (for instance about a user or region) with each request, set the extra body parameter like so:
client.chat.completions.create(
model="router",
messages=[
{"role": "user", "content": "Write small poem"},
],
extra_body={
"models": ["gpt-3.5-turbo", "claude-2.1"],
"max_cost": 0.02,
"willingness_to_pay": 0.01,
"extra": {
"ip": "123.123.123.123",
"Timezone": "UTC+0",
"Country": "US",
"City": "New York",
}
},
)
Last updated