Chat completions

Request

Plain messages serialize exactly as they do for OpenAI. Every field the OpenAI chat completions API documents is accepted: forwarded when every provider accepts it, mapped when Inferbase has an equivalent, or accepted and reported as not forwarded.

A field not in the OpenAI specification and not an Inferbase extension is a 422 naming it.

The routing object

All fields optional. A request may narrow what its routing policy allows and never widen it; see Routing policies.

Timeouts

Each attempt has a first-token budget and the request has a total deadline; the routing policy sets both. On a streamed request the budget is measured to the first token, not to completion, so a long answer is never cut off for being long. On a non-streamed request the first token arrives with the whole answer, so each attempt is given at least 60 seconds and the request at least 90. When the model thinks, a non-streamed answer is not cut for its length: it fails only when the model stops sending for 30 seconds, or when the whole request passes 10 minutes. For long reasoning, stream the response: a stream with nothing to send for 15 seconds receives an SSE comment line, : processing, so proxies and idle timeouts keep the connection open. SSE readers ignore comment lines. A conversation mid tool loop falls back only within the model that made the call.

Structured output

response_format takes {"type": "json_object"} for JSON, with the prompt also asking for JSON, or {"type": "json_schema", "json_schema": {...}}, forwarded to the provider as written. Routing serves only models with verified structured-output support.

When a non-streaming answer arrives wrapped in code fences or commentary, the JSON is taken out of it and usage.structured_output_repair says so. Nothing is rewritten to make it parse. The routing policy can turn extraction off, or have the model asked once more when there is no document to take.

Example

curl https://api.inferbase.ai/api/v1/inference/chat/completions \
  -H "Authorization: Bearer inf_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is machine learning?"}
    ],
    "temperature": 0.7,
    "max_tokens": 256,
    "routing": {"optimize": "quality"}
  }'

Response

The OpenAI shape. Two fields carry Inferbase meaning.

• model is the model that served the request, by its catalog id (slug, or slug:variant for one variant). It never says auto, and it never names the provider.
• id is the request id, the handle for GET /routing-decisions/{request_id}.

Non-streaming responses also carry what the request cost your organization and how long it took, as usage.cost_micros (micro-dollars: the provider price, plus any guardrail checks, plus the managed markup; on your own provider key only the guardrail checks are charged, and the free plan is charged nothing) and usage.latency_ms (wall time). This is the amount charged for the request, so it matches Usage. For a brokered tool run both figures cover the whole run, and the same two in the X-Inferbase-Cost-Micros and X-Inferbase-Latency-Ms headers. A stream's closing usage event carries cost_micros, latency_ms and ttft_ms. Non-streaming responses also carry the served model in the X-Inferbase-Model header, and the routing decision as a routing field: the same object the routing event carries on a stream, so a client that does not stream still sees the task, the pick, its decisiveness and the ranked alternatives.

A reasoning model's thinking arrives as message.reasoning_content, beside content; a streamed answer carries the same text on delta.reasoning_content. Thinking counts toward max_tokens and is billed as output, so a budget spent reasoning ends with finish_reason: "length", reasoning present and no content, or with an answer cut short; the routing object then says truncated: "max_tokens" (on a stream, the final usage event does). Raise max_tokens, or send extended_thinking: false for a direct answer. Some models reason whatever the request says (gpt-oss, on every provider we serve it from): for those no switch is sent, the routing object says thinking: "on" with thinking_reason: "model_ignores_off", and only a larger max_tokens makes room. On model: "auto", such a model is placed behind the others in the chain when max_tokens leaves too little room for the reasoning it usually does, and the routing object's reasoning says so; a model you name is always used. On a non-streamed request, an answer cut off before any content is treated as a failed attempt and the next model in the chain is asked. On a streamed request with extended_thinking: false, reasoning is held until the first content or tool call, so an answer that ends with reasoning only is a failed attempt in the same way, and the held reasoning is sent ahead of the content; the first token arrives with the answer. With thinking left on, reasoning streams as it is produced. When the message is sent back as part of the next request, its reasoning_content is accepted and not forwarded to the model.

{
  "id": "req-9f2c1e...",
  "object": "chat.completion",
  "created": 1712345678,
  "model": "openai-gpt-oss-120b",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Machine learning is a subset of artificial intelligence..."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 24, "completion_tokens": 128, "total_tokens": 152}
}

usage may carry dropped_params, structured_output_repair, cache token counts, and, for brokered tool runs, a broker block.

Streaming

With stream: true the response is server-sent events: OpenAI-shaped chunks wrapped by three Inferbase events in a fixed order.

1. A routing object, before any content, carrying the decision. See Routing.
2. The chunks. delta.reasoning_content carries thinking passthrough, delta.tool_calls carries tool call fragments, and the last chunk carries finish_reason. content_filter means the model declined to answer (a provider's safety refusal); the answer may be empty, and it is not retried on another model.
3. A usage event, then [DONE].

A failure after the stream has started arrives as an event shaped {"object": "error", "error": {...}} followed by [DONE], with the routing disposition in code. Clients that do not know the Inferbase events skip them.

Errors

See Errors for the full list and the envelope.

Last updated October 12, 2026.