LLM routing that picks the best model for the task
Send model="auto": each request is classified for its task, narrowed to the models that clear the quality bar, and served by the one that wins on your objective.
The real classifier and engine over the managed pool, no model called. In production the same decision arrives before the first token and is kept by request id.
What LLM routing does
The decision of which model should answer a request, made per request on what the request asks for, rather than once when the code was written.
An LLM router reads the prompt, works out what kind of task it is and how demanding, and picks from the models that are good enough for it on what you care about most: cost, quality, latency or a balance. Without a router, every request goes to the one model someone chose at design time, and the bill is set by the hardest request in the workload rather than the actual one.
Inferbase runs the router inside the gateway, so the decision is served, not recommended. Each pick rests on benchmark evidence for the task, an unmeasured model is treated as unknown rather than assumed, and the decision is returned with the response and kept by request id. Routing is one of the four capabilities behind the gateway, beside the catalog, security and observability.
Inferbase Gateway
One OpenAI-compatible API, with failover, rate limits and budgets behind it.
- knowledge
Catalog
Curated creators, verified variants, benchmark evidence where it exists, live prices.
- decision
Routing
A prompt classified, a model chosen on your objective, the decision kept.
- policy
Guardrails
Which models may serve, guardrails at three doors, which tools an agent may call.
- record
Observability
Decision, verdicts, tokens, cost and latency per request; usage and an audit log.
What LLM routing saves
Routing sends every request to the right-sized model for the task, which is what turns a flat frontier-model bill into one that follows the actual work.
23 more prompts went to five other models, the priciest of them held back for the hardest generation work. 10 stayed on the baseline, where nothing cheaper cleared the quality bar.
Every rate above is what that model costs per million tokens on the route that served it. What a workload saves depends on what it was paying before, which is a question about your bill rather than ours.
Measured on 100 mixed-difficulty prompts, 2026-07-06. Quality equivalence is projected from benchmark evidence, not judged output by output. We publish no savings percentage: what a workload saves depends on what it pays today, which Routing Preview measures on your own traffic.
How LLM routing works
Three stages before the first token: classify the task, score the models that qualify on your objective, serve the winner with two fallbacks ready.
“Compare RAG and fine-tuning for a support bot.”
- task
- analysis
- complexity
- high
- objective
- balanced
- DeepSeek V4 Pro0.91
- Qwen 3.5 122B0.84
- Llama 3.1 70B0.78
ranked on your objective
streaming
- fallback
- Qwen 3.5 122B
- your code
- model="auto"
One prompt, four LLM routing objectives
The objective you set changes which model wins on the very same request. Set it on the key, override it on any request.
“Summarize this 20-page provider contract and flag risky clauses.”
Highest analysis score of the eligible models. Worth the spend when a missed clause is expensive.
Only models that clear the bar for the task
Price comes after quality, never instead of it. Three presets set how high the bar sits; the evidence is benchmark scores for the task.
Three presets
Strict, standard and permissive set the floor on the task score. A preset that leaves no qualifying model relaxes one step to the next.
Unknown is unknown
A model with no measurement for the task does not qualify on guesswork, and a variant is never assumed to match the model it was derived from.
Context fit
A candidate has to fit the request's tokens in its servable window; the router checks the window the host actually serves, not the advertised one.
Structured output
A request with a JSON schema goes only to a model whose deployment honors it; tool calls only to a model that supports them.
The fainter marks are the other two floors, permissive lower and strict higher. Price is compared among the models marked in; a preset that leaves nobody in relaxes one step, and the relaxation is on the decision record.
Turn on LLM routing with one line
Keep the OpenAI SDK, your prompts and your request shape. Point the base URL at Inferbase and set the model to auto; name a model again to pin it.
- Your OpenAI SDK
- Your prompts and messages
- Streaming
- Request and response shape
- base URLapi.inferbase.ai/api/v1/inference
- modelgpt-4oauto
Two values. That is the whole integration.
Every LLM routing decision explained
Every route leaves a record you can read: the task, the candidates, their scores, the winner and how clearly it won.
Every decision is auditable
The task, the eligible models, their scores, and the winner are all on the record. Nothing happens in a black box.
Unknown is an honest answer
A model we have not evaluated for a task is marked unknown, never assumed as good as the model it came from.
You stay in control
Set the objective, scope the eligible models, or pin one outright. Routing executes; the choice is yours.
Routing decides; the scope is yours
Every control lives on the API key and changes without a deploy: what the router may choose from, how high the bar sits, when to stay put.
Pin a model
Name a model instead of auto and it is served as named, with the same receipts, limits and failover and no routing fee.
Model pool
A list of catalog models the router chooses from and nothing else, per key.
Scope
The managed catalog, only what your own connected provider keys reach, or both. A model reachable both ways runs on your key when you asked for your keys.
Session affinity
A conversation stays on the model that already holds it unless a switch is worth the cost. On by default; off per request.
Per request
The objective and the session are set on the key and can be overridden on any single request.
Routing preview
Try prompts on the card above with no model called, or upload past traffic in the dashboard and see what the router would have chosen.
- model
- autoper request
- objective
- balancedper request
- quality preset
- standard
- scope
- managed catalog + your keys
- model pool
- any qualifying model
- session affinity
- onper request
Changed in the dashboard or the API; the next request uses the new settings, no deploy.
How LLM routers compare
Routers recommend, proxies follow your rules, frameworks make you host it. Inferbase decides, serves and keeps the record, in one API.
| Inferbase | OpenRouter | LiteLLM | Portkey | NotDiamond | RouteLLM | |
|---|---|---|---|---|---|---|
| Picks the best model per request | First-party | Via NotDiamond add-on | Beta tiers you map by hand | No, rules you define | Yes | Strong vs weak only |
| Routes and serves in one API | Yes | Yes | Proxies via your providers | Proxies via your providers | No, you run it | No, self-hosted |
| Per-request decision audit | Yes | No | Logs and cost tracking | Deep logs and traces | Recommend-side | Build your own |
| Nothing to self-host or calibrate | Yes | Yes | No | Hosted, rules are yours | Yes | No |
| Model breadth | Curated catalog of open models | Hundreds of models | 100+ providers, your keys | 1,600+ models, your keys | Your chosen pool | Two models |
Frequently asked questions about LLM routing
What LLM routing is, how it differs from a gateway, a load balancer and a fallback, and how this router decides.
LLM routing is the decision of which model should answer a request, made per request rather than once when the code is written. A router reads the prompt, works out what kind of task it is and how demanding, and picks from the models that are good enough for it on whatever you care about most: the cheapest, the strongest, the fastest, or a balance. Without it, every request goes to the one model someone chose at design time, whether or not it needs that model.
In three stages before the first token. The request is classified for the kind of work it is and for its complexity. The catalog is narrowed to the models that clear a quality bar for that task, on benchmark evidence, and that fit the request: its context length, its tools, its output format. The survivors are scored on your objective and the winner is served, with two fallbacks ready. The decision is kept with the request.
A router decides which model should answer; a gateway executes the request and governs it with keys, limits, budgets, logs and failover. Some routers only recommend a model and leave the call to you; some gateways only proxy to the model you named. Inferbase runs the router inside the gateway, so the decision is served rather than recommended, and the record of it sits beside the usage receipt.
Load balancing spreads identical requests across replicas of the same thing, so that no one of them is overloaded; it does not look inside the request. LLM routing looks inside: two requests that arrive at the same endpoint can go to different models because they ask for different things. Routing can sit on top of a balanced pool, and on Inferbase it does, since a chosen model may be served by more than one host.
Fallback reacts to a failure: the first model errors or times out, so a second one is tried. Routing decides up front, before anything is called, which model should go first. Inferbase does both. Every routed request carries a primary and up to two fallbacks in the order the router ranked them, and if the primary fails on a transient or server-side error the next is tried; a client error such as an invalid request is returned as it is.
When a workload mixes tasks and difficulties, which most production traffic does: a summarisation call does not need the model a hard coding task needs, and paying for it anyway is the cost routing removes. We publish no savings percentage: one was withdrawn in September 2026 because the baseline price it was measured against is no longer one we can substantiate, and a ratio is only as good as its denominator. What routing does is send each request to the cheapest model that clears the quality bar for it, so the gain is the gap between what you pay now and what the right-sized model costs. A workload that is all one hard task gains less; the page above shows the run and its caveats, and Routing Preview measures it on your own traffic.
On evidence, then on your objective. Each model carries benchmark scores per task family from a public, cited source; a model with no measurement for the task is treated as unknown and does not qualify on guesswork, and a variant is never assumed to match the model it was derived from. Among the models that clear the bar for the task and fit the request, the objective ranks them: cost prefers the cheapest, quality the strongest, latency the fastest measured, balanced a weighting. The pick, the runners-up and how clearly it won are all on the record.
Four objectives: cost, balanced, quality and latency. The objective is set on the API key and can be overridden on any single request, so an application can run balanced by default and ask for quality on the calls that matter. The same prompt can produce a different pick under each objective; the switch on this page shows that on one request.
Yes, in three ways. A key can carry a model pool, a list of catalog models the router chooses from and nothing else. A key carries a preset (strict, standard or permissive) that sets how high the quality bar sits; a preset that leaves no qualifying model relaxes one step to the next. And a key carries a scope: the managed catalog, only what your own connected provider keys reach, or both. To take the router out of the loop entirely, name a model and it is served as named.
About a tenth of a second at the median on typical prompts and more on very long ones, measured in July 2026; most of that is the classifier. When you stream, the decision arrives as the first event, before any content token. The time spent is reported on every routed response as routing_time_ms, so you can read your own figure rather than ours.
Not unless switching is worth it. A conversation arrives as the same prefix plus one more message each turn, and moving it forfeits the host's prompt cache and changes the model's style or tool behavior mid-flight. Session affinity keeps turns that share a session on the model that already holds the conversation; you can send an explicit session id (routing.session, or the X-Inferbase-Session header from an Anthropic SDK) or let turns be recognized by their shared opening. It is on by default for routed requests and can be turned off per request.
Every routed request writes a decision record: the task the classifier saw, its complexity, whether the estimate was a heuristic fallback, the objective, how many models qualified, which won, its decisiveness, and each fallback attempt if any. When streaming it is the first event on the stream; afterwards it is retrievable by request id from the API and in the dashboard's activity log. Nothing happens in a black box.
One line. Routing runs behind the same OpenAI-compatible API: point an existing OpenAI SDK at the Inferbase base URL and send model="auto". Chat completions, streaming, tool calls and structured output pass through unchanged, and Anthropic-shaped clients are served on the Messages endpoint and route the same way. Switch back to a pinned model at any time by naming it instead of auto.
Yes. Connect your OpenAI, Anthropic, Gemini or other provider accounts to a project and set the key's scope to your keys, or to both; the router then chooses among the cataloged models your keys reach, on the same evidence, and the tokens are billed by your provider. A model reachable both ways runs on your key when you asked for your keys. Models your keys can reach but the catalog has not measured are not routed to; they are served only when you name them.
Free during beta. The published plan meters routing decisions at $1 per 1,000, with the first 5,000 each month free; a request that names a model is never charged a routing fee. Tokens are billed separately: on the managed catalog at the host's price plus a flat 6%, or by your own provider when you route on your keys. New accounts start with $5 of credit.
Put intelligence in the middle.
One gateway in front of every model, with your policies applied and every decision on record. Start with $5 of credit and 5,000 routing decisions a month, no card required.