When model is auto (or inferbase/auto, accepted as the same word on every endpoint), Inferbase classifies the request, keeps the models in the policy's pool that clear the evidence bar for that task and fit the request, ranks them on the objective, and serves the winner with fallbacks ready. The pool, objective, and scope come from the routing policy the key runs under: the policy attached to the key, else the project default, else the organization default, else the Inferbase default. Individual requests need no routing fields. See How routing works.
Per-request narrowing is done with the routing object on chat completions.
The routing event
On a streamed response the first event, before any content, is the decision. A non-streaming response carries the same object, minus object, as its routing field.
text
data: {"object": "routing", "task": "coding", "complexity": "easy", "estimated": false, "decisiveness": 0.95, "decisiveness_basis": "lead", "forced_single": false, "model": "openai-gpt-oss-120b", "measured": true, "optimize_mode": "balanced", "routing_time_ms": 112}| Field | Meaning |
|---|---|
task | The task the request was classified as. |
complexity | How demanding the request is: easy, standard or hard. |
estimated | true when classification fell back to a heuristic estimate. |
model | The model chosen, by its catalog id (slug, or slug:variant for one variant). |
measured | false when the pick has no benchmark evidence for the task. |
optimize_mode | The objective in force. |
decisiveness | The winner's margin over the runner-up model on the objective's axis, 0 to 1. 0 is an exact tie. null when there is no margin to state; decisiveness_basis says why. |
decisiveness_basis | lead when the margin was measured against another model. unmeasured when a second model competed but one side has no value on the axis (no price, no latency sample, no benchmark cell). no_contest when no second model qualified. pinned when the request named the model. |
forced_single | true when only one model qualified, so a forced pick never reads as a decisive win. |
routing_time_ms | Time spent choosing, measured on this request. It scales with prompt length; read the value your own traffic returns. |
session | Whether the conversation stayed on its model, and why. |
cache_fit | Whether a prompt-cache preference was satisfied, and how the chosen provider caches. |
policy | Which routing-policy rules changed the decision. Absent when no policy applied. |
proxied | true when the pick had no benchmark for the task and was ranked on its overall score instead. |
base_id | The catalog id of the chosen model's base, stable across its variants and providers. |
variant_label | The variant served (for example fp8 or turbo); null for the base model. |
variant_substituted, requested_variant | For a variant named in model or model_pool with no servable route: true, and the variant that was asked for. |
slow_tier | true when the model is known to start slowly, so the first token may take longer. |
thinking, thinking_reason | Whether the chosen model runs with thinking on or off for this request. thinking_reason appears when auto decided or when the outcome is not what the request asked: max_tokens, easy_prompt or latency_mode when auto turned thinking off (the output limit leaves no room to think before answering, the prompt is among the easiest it classifies, or the objective is latency); task_needs_thinking when auto asked for it because the task is a factual question, maths or coding; cheaper_with_thinking when auto left thinking to the model because answering without it would have cost more; model_ignores_off when thinking was off on the request (yours or auto's) but the model reasons regardless, such as gpt-oss, so its reasoning counts toward max_tokens; route_cannot_think when you asked for thinking and the model, as served, cannot reason. Absent only when the mode is not known (a model whose behaviour we have not measured yet, with thinking left to its default). The reasoning the answer actually spent is in usage.completion_tokens_details.reasoning_tokens where the provider states it, and on every request in Logs. |
expected_wait_ms, wait_basis | The latency the chosen model was ranked on for this request, in milliseconds, and which wait it is: first_token (a streamed request where the model does not think), first_answer_token (a streamed request where it thinks: its usual reasoning comes first), or end_to_end (a request that is not streamed: nothing arrives before the whole answer). Measured per route from recent traffic at Inferbase; null when the route has not been measured yet. |
truncated, truncated_hint | Present only when the answer stopped at max_tokens while the model was thinking: truncated is "max_tokens", and truncated_hint says what to change. Thinking counts toward max_tokens, so the visible answer got less than the limit you set; the limit is never raised for you. Raise max_tokens, or send extended_thinking: false for a direct answer. On a streamed request it is on the final usage event, since the routing event is sent before the answer. |
reasoning, alternatives | The rule that decided, and the contenders it was chosen from with their standings. |
dropped_pool_ids | Models named in the pool that could not be used for this request. Absent when none were dropped. |
classification, classification_cached | pending when classification ran after the pick (task and complexity are then null); true when the classification came from cache. Each is absent otherwise. |
OpenAI SDKs skip events they do not recognize; read the raw stream to consume it in band.
Routing decisions
| Method and path | GET /routing-decisions/{request_id} |
| Purpose | The persisted decision for one routed request: classification, the eligible pool, the fallback chain, decisiveness, and the per-step eligibility trail. |
| Authentication | API key or session |
request_id is the id of the chat completion response. Log it beside your own records and fetch the decision whenever you need it.
Response, abridged
json
{
"request_id": "req-9f2c41d8a6b34e17",
"routing_mode": "balanced",
"classifier_task_family": "coding",
"complexity_band": "hard",
"eligibility_pool_size": 12,
"final_disposition": "served",
"decisiveness": 0.73,
"forced_single": false,
"chain_attempted": ["<route id>", "<route id>", "<route id>"],
"route_displays": {"<route id>": "GPT OSS 120B"},
"filter_trail": [
{"step": "pool_loaded", "in": 20, "out": 20},
{"step": "context_fit", "in": 20, "out": 16},
{"step": "preset_eligibility", "in": 16, "out": 12, "preset": "standard"},
{"step": "chain", "in": 12, "out": 3}
],
"filter_trail_text": [
"20 models in the pool.",
"16 of 20 models fit the request's length.",
"12 of 16 models meet the standard quality floor for this task.",
"3 models in the final chain: the pick, then its fallbacks."
]
}| Field | Meaning |
|---|---|
chain_attempted | Routes in fallback order; the first is the pick. The pick's model on its other providers comes next, then other models. route_displays maps each to its catalog model name. |
filter_trail | How the candidate pool narrowed at each stage, from every servable route to the final chain. |
filter_trail_text | The same steps in plain words, one sentence each, in the same order. Read this one to show or explain a decision; the records above carry the numbers for an audit. |
final_disposition | served, or how the request ended otherwise. |
The same record is shown in the portal under Observability › Logs, rendered with model names. See Reading a routing decision.
Models
| Method and path | GET /models |
| Purpose | The models available for inference. The first entry is auto, so routing is selectable in any client that builds its model picker from this list. With a key, the project's provider-key models are appended, and when the key's policy requires a pinned model to be in its pool, only the models that pool admits are listed. |
| Authentication | None |
json
{
"object": "list",
"data": [
{"id": "auto", "object": "model", "owned_by": "inferbase", "name": "Inferbase Auto - Smart Routing"},
{"id": "openai-gpt-oss-120b", "object": "model", "owned_by": "inferbase", "name": "GPT OSS 120B"}
]
}Model health
| Method and path | GET /models/{model_id}/health |
| Purpose | Whether the provider that serves the model is reachable. |
| Authentication | None |
json
{"model": "openai-gpt-oss-120b", "status": "ready"}status is ready, loading, or unavailable. It is a liveness probe against the provider's API, not a cold-model detector: a warm and a cold model both report ready, and loading means the provider's API is not answering. Polling it does not warm anything.
Last updated October 12, 2026.