How to Route OpenAI SDK Calls Across Multiple Models

Most inference migrations fail to start because teams imagine them as rewrites. The reality of moving to an OpenAI-compatible routing layer is closer to a configuration change: the SDK stays, the message shapes stay, the streaming handlers stay, and what changes is where the client points and how much thought each call site gives to the model parameter. The work worth planning is not the mechanical swap but the decisions around it, which call sites should auto-route, which should stay pinned, and how you verify behavior after cutover.

This guide walks that migration end to end against Inferbase, which serves an OpenAI-compatible API in front of a routed, multi-provider model pool. The shape generalizes: retrieval, tools, memory, and orchestration stay in your stack, and only the generation call crosses the new boundary. If you want the conceptual background on routing itself first, what is model routing covers it; this article assumes you have decided to try it and want the working steps.

The two-line change

The client construction is the entire mechanical migration. Python:

from openai import OpenAI

client = OpenAI(
    api_key="inf_your_api_key",
    base_url="https://api.inferbase.ai/api/v1/inference",
)

response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this ticket for the on-call engineer: ..."}],
)
print(response.choices[0].message.content)
print(response.model)  # the model that actually served this request

Node.js:

import OpenAI from 'openai'

const client = new OpenAI({
  apiKey: 'inf_your_api_key',
  baseURL: 'https://api.inferbase.ai/api/v1/inference',
})

const response = await client.chat.completions.create({
  model: 'auto',
  messages: [{ role: 'user', content: 'Summarize this ticket for the on-call engineer: ...' }],
})
console.log(response.choices[0].message.content)
console.log(response.model)

Keys are created in the dashboard and carry the inf_ prefix. Everything else in these snippets is the OpenAI SDK you already run. The one detail worth pinning to memory is the base URL: it is https://api.inferbase.ai/api/v1/inference, and the SDK appends /chat/completions itself.

Inventory your call sites before you flip anything

A codebase of any age accumulates LLM call sites with different stability requirements, and the productive first step is listing them. For each call site, answer two questions: does the output feed a machine or a human, and does request difficulty vary. Those two answers determine the model parameter.

The pattern that emerges in practice: the highest-volume surfaces are usually the heterogeneous, human-facing ones, and those are exactly where routing earns its keep, because the easy majority of requests stops paying frontier prices. The machine-facing paths are fewer, lower-volume, and should not be routed at all in the first pass. Migrating them is not wrong; it is just a separate, later decision with its own verification.

Pinning: the same discipline you should already have

Pinning a call site means passing a concrete model id instead of auto:

response = client.chat.completions.create(
    model="Qwen/Qwen3-14B",
    messages=[{"role": "user", "content": "Extract the invoice fields below as JSON: ..."}],
    temperature=0.0,
)

The valid ids are whatever GET /models on the API returns, and they are the upstream serving ids, not marketing names. Two practices from the wider ecosystem apply with full force here. First, pin the most specific identifier available; we wrote up what happens when teams ride floating aliases and the short version is that a pinned dated identifier is the only version contract a provider actually honors. Second, record the pin decision next to the prompt it protects, because the pin exists to keep that prompt's tuned behavior stable, and whoever revisits it in six months needs to know those two things travel together.

A pinned call through a routing layer still buys something over a direct provider call: if the pinned model is served by more than one provider, failover between them happens behind the same request rather than in your retry logic. The fallback chain mechanics apply to pinned traffic too; pinning constrains which model serves, not which provider survives an outage.

Auto-routing: pool first, then mode

Before any call site goes to auto, curate the model pool. The pool is the list of model ids the router is allowed to pick from, set on the API key or overridden per request, and it converts "the router picked something" from a trust question into a policy question. If your organization has models it cannot send data to, they are simply not in the pool, and the router cannot select them.

response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Does this clause conflict with the attached policy? ..."}],
    extra_body={
        "routing": {
            "optimize": "quality",
            "model_pool": [
                "openai/gpt-oss-120b",
                "Qwen/Qwen3-235B-A22B-Instruct-2507-tput",
                "Qwen/Qwen3-14B",
            ],
            "min_context_window": 32000,
        }
    },
)

The optimize axis takes exactly four values: cost, quality, latency, or balanced. Set the default on the key and reserve the per-request override for surfaces that genuinely differ; a support widget optimizing for latency while a nightly summarization job optimizes for cost is the natural shape. The routing object also accepts min_context_window and max_cost_per_1m_tokens when a call site has hard constraints that the mode alone does not express.

One honest note on what routing optimizes today: model quality evidence is general per-task benchmark data, not your domain's own evaluations. Routing on your own eval results is where this category is heading, but claiming it today would be overclaiming. What you get now is the cost and latency arbitrage across a pool you approved, with quality guarded by the pool's floor: nothing in the pool is a model you were unwilling to serve.

Verify the cutover by reading the routing metadata

The migration is not done when requests succeed; it is done when you can see what the router is doing. Every response discloses the served model: non-streaming in the model field and the X-Inferbase-Model header, streaming in a routing event that arrives before the first content token:

{
  "object": "routing",
  "task": "summarization",
  "complexity": "simple",
  "model": "Qwen/Qwen3-14B",
  "optimize_mode": "balanced",
  "decisiveness": 0.83,
  "routing_time_ms": 11,
  "alternatives": [
    { "model_id": "openai/gpt-oss-120b", "quality_score": 0.78, "cost_score": 0.41, "latency_score": 0.66 }
  ]
}

Log this event keyed to your own request id. In the first week after cutover, three checks matter. Watch the distribution of served models per surface: a chat surface on balanced should show a spread, and a spread of one model usually means the pool is too narrow or the traffic more uniform than assumed. Watch decisiveness, which reports how clear-cut each pick was over the runner-up: persistently low values on a surface mean the pool contains near-equivalents and could be simplified. And watch the terminal usage event on streamed responses, which carries tokens, latency, and cost in microdollars per request, because that is the ledger your before-and-after cost comparison should be built from, not the invoice at month end.

data: {"object": "usage", "model": "Qwen/Qwen3-14B", "prompt_tokens": 412,
       "completion_tokens": 128, "total_tokens": 540, "cost_micros": 187,
       "latency_ms": 940, "ttft_ms": 210}

Map the error surface into your existing retry logic

Your application already has retry and error-handling paths built around one provider's failure modes. The routed endpoint uses standard HTTP semantics, so the mapping is mostly mechanical, but two of the codes carry different operational meaning than you may be used to and deserve explicit branches.

The two worth internalizing: a 402 is a billing state, not an incident, and paging an on-call engineer for it wastes a page (route it to whoever owns the account); and a 404 on a pinned call is the silent-swap problem arriving loudly, meaning the id you pinned has left the catalog, which is a migration task rather than a retry candidate. Everything retryable is worth one attempt against the same endpoint before your own fallback logic engages, because provider-level failover already happened inside the request; if a 502 reaches you, the layer's own alternatives were exhausted.

import time
from openai import APIStatusError

RETRYABLE = {429, 502}

def call_with_retry(make_request, attempts: int = 3):
    for attempt in range(attempts):
        try:
            return make_request()
        except APIStatusError as e:
            if e.status_code not in RETRYABLE or attempt == attempts - 1:
                raise
            time.sleep(2 ** attempt)

Migrate your streaming handler for the two new events

A routed streaming response is an OpenAI-style SSE stream with two additions: a routing event before the first content chunk, and a usage event after the last one. Both carry an object field that is not chat.completion.chunk, which matters for how your parser treats them.

If you consume the stream through the OpenAI SDK, the safe pattern is to treat any event without choices as metadata rather than content. The SDK constructs these events as chunk objects without populating the fields it does not recognize, so guard on the field rather than assuming every event is a content delta:

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "..."}],
    stream=True,
)

for chunk in stream:
    if not getattr(chunk, "choices", None):
        # routing or usage metadata, not a content delta
        continue
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="")

If you want the metadata itself, which you should for the audit trail, parse the SSE stream directly. The routing event arrives first and tells you the served model before the first token; the usage event arrives last and carries the per-request cost:

import json
import httpx

with httpx.stream(
    "POST",
    "https://api.inferbase.ai/api/v1/inference/chat/completions",
    headers={"Authorization": "Bearer inf_your_api_key"},
    json={"model": "auto", "messages": [{"role": "user", "content": "..."}], "stream": True},
    timeout=120,
) as response:
    for line in response.iter_lines():
        if not line.startswith("data: ") or line == "data: [DONE]":
            continue
        event = json.loads(line[6:])
        if event.get("object") == "routing":
            log_routing_decision(event)   # served model, task, complexity, decisiveness
        elif event.get("object") == "usage":
            log_request_cost(event)       # tokens, cost_micros, latency_ms, ttft_ms
        elif event.get("choices"):
            handle_content_delta(event)

Most teams keep the SDK for the application path and run the raw parser only where the metadata gets logged, which is usually a single wrapper module rather than every call site.

Troubleshooting the first hour

Why does every request return 401?

Almost always the base URL rather than the key: the OpenAI SDK appends /chat/completions to whatever base_url you set, so the value must be https://api.inferbase.ai/api/v1/inference exactly, with no trailing path segments. The second most common cause is an environment still carrying the old provider's key; the key must start with inf_.

Why does my pinned model return 404?

The model value must be an id the API's GET /models currently returns, which is the upstream serving id, not a marketing name or a catalog page slug. List the endpoint and copy the id exactly; ids are case-sensitive.

Why did a long prompt fail immediately?

The request reserves capacity against max_tokens plus the prompt before serving. If the combination exceeds every eligible model's context window, the request fails up front rather than mid-generation. Either raise routing.min_context_window so the router only considers models that fit, or reduce max_tokens to what the response actually needs.

Why do responses name a model I did not choose?

That is the disclosure working: auto picked it, and the model field plus the routing event tell you what and why. If a specific surface must never vary, that surface should pin, which is precisely the auto-versus-pin decision from the inventory table above.

Boundaries to know before cutover

Three boundaries to respect, stated plainly rather than discovered in production. Native tool calling (tools / tool_choice) and structured output (response_format with json_object or a forwarded json_schema) are both part of the chat endpoint, with one constraint: tool-carrying requests must pin a concrete model, and auto with tools returns a structured 400 until routed tool support ships; our coding-agent guide covers wiring a full tool loop, and structured outputs covers choosing between the two mechanisms. Requests are capped at 128K max_tokens and 200 messages. And prompt or response content is not stored on the serve path at all; the only content stores are two opt-in features (saved playground conversations and eval uploads), and an account-level zero-retention flag disables even those, which is the setting to flip before any regulated surface migrates.

How to run the migration, step by step

1. Create a key per surface, each with its own optimize_mode and pool, so behavior and spend attribute cleanly from day one.
2. Migrate one pinned, low-stakes call site first. This validates auth, the base URL, and your logging without touching behavior, since a pinned call serves the same model as before.
3. Move your highest-volume heterogeneous surface to auto behind a percentage rollout if you have one, comparing served-model distribution, latency, and the per-request usage events against the incumbent.
4. Leave machine-facing pinned paths for a second pass, migrated one at a time with their own output-contract checks.
5. Keep the rollback trivial: the old client construction is two lines away, which is the entire point of an OpenAI-compatible boundary.

The API reference has the full parameter surface, and the model catalog is where pool candidates get evaluated. The migration itself is an afternoon; the inventory and the verification discipline around it are what make it stick.