Inference API Reference

Most inference endpoints require authentication (the exceptions are marked below: the model listing, the health check and the routing preview are open). You can use either an API key (for programmatic access) or a JWT token (for browser-based access).

API Key (recommended for code)

API keys start with inf_ and are passed in the Authorization header.

Authorization: Bearer inf_your_api_key_here

JWT Token (browser sessions)

If you're already logged into Inferbase, the JWT cookie is sent automatically. No additional setup needed for dashboard interactions.

Request Body

curl -X POST https://api.inferbase.ai/api/v1/inference/chat/completions \
  -H "Authorization: Bearer inf_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-4.7-Flash",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is machine learning?"}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Response

{
  "id": "req-9f2c1e...",
  "object": "chat.completion",
  "created": 1712345678,
  "model": "zai-org/GLM-4.7-Flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Machine learning is a subset of artificial intelligence..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 128,
    "total_tokens": 152
  }
}

OpenAI-compatible embeddings from the served embedding catalog. Embeddings are pinned-model only: switching embedding models between requests breaks your vector index's consistency, so there is no "auto" here. Billed on input tokens.

curl -X POST https://api.inferbase.ai/api/v1/inference/embeddings \
  -H "Authorization: Bearer inf_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-Embedding-4B",
    "input": ["first text to embed", "second text to embed"]
  }'

Response follows the OpenAI shape: data carries one embedding vector per input, in input order, with input-token usage. dimensions and user are forwarded to the vendor when set; unknown fields are rejected with a 422, as on chat completions.

InferRoute is the classification and scoring engine behind smart model routing. When you set model: "auto", InferRoute analyzes the prompt for task type and complexity, scores each model in your pool on fit, cost, and latency, and routes the request to the highest-ranked option. The model pool and optimization mode are configured on your API key in the dashboard, so individual requests require no additional parameters.

How it works

1. Create an API key and select which models to include in the pool
2. Choose an optimization mode: Balanced, Quality, Cost, or Speed (API values balanced, quality, cost, latency)
3. Send requests with model: "auto"
4. InferRoute classifies the prompt by task type and complexity, then selects the highest-scoring model

curl -X POST https://api.inferbase.ai/api/v1/inference/chat/completions \
  -H "Authorization: Bearer inf_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Write a Python function to sort a list"}]
  }'

Per-request overrides

The key-level defaults apply to every request automatically. To override them for a specific call, pass a routing object in the request body.

curl -X POST https://api.inferbase.ai/api/v1/inference/chat/completions \
  -H "Authorization: Bearer inf_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Analyze this contract..."}],
    "routing": {
      "optimize": "quality",
      "model_pool": ["zai-org/GLM-4.7-Flash", "openai/gpt-oss-120b"]
    }
  }'

Streaming routing event

When streaming is enabled with smart routing, the first SSE event contains the routing decision before any content tokens are sent:

data: {"object": "routing", "task": "code_generation", "complexity": "simple", "decisiveness": 0.95, "model": "zai-org/GLM-4.7-Flash", "routing_time_ms": 112}

routing_time_ms is the time spent selecting the model, measured per request and reported on every routed response. It scales with prompt length, so read the value your own traffic returns rather than treating the figure above as a guarantee.

Every routed request writes a persistent decision record: what the classifier saw, which models qualified, what was picked over what, and how clear-cut the pick was. The record is yours to query, so routing never has to be taken on faith. There are three ways to read it.

1. Inline, on streaming responses

The routing SSE event above delivers the decision before the first content token. Standard OpenAI SDKs skip events they do not recognize, so read the raw stream if you want it in-band.

2. By request ID, after the fact

Every chat completion response carries a request ID in its id field (req-...). Log it alongside your own request records, then fetch the full decision whenever you need it, with the same API key.

{
  "request_id": "req-9f2c41d8a6b34e17",
  "routing_mode": "balanced",
  "classifier_task_family": "code_generation",
  "classifier_complexity": 0.41,
  "eligibility_pool_size": 12,
  "final_disposition": "served",
  "decisiveness": 0.73,
  "forced_single": false,
  "chain_attempted": ["<route-id-1>", "<route-id-2>", "<route-id-3>"],
  "route_displays": {"<route-id-1>": "GLM-4.7-Flash", "<route-id-2>": "gpt-oss-120b"},
  "filter_trail": [
    {"step": "pool_loaded", "in": 20, "out": 20},
    {"step": "context_fit", "in": 20, "out": 16},
    {"step": "preset_eligibility", "in": 16, "out": 12, "preset": "standard"},
    {"step": "chain", "in": 12, "out": 3}
  ]
}

3. In the dashboard

On the Usage page, any smart-routed request in the activity log expands to the same record, rendered with model names, so a human can review routing without touching the API.

API keys carry routing configuration (model pool and optimization mode) so your app does not need to send these on every request. Manage keys in the dashboard or via these endpoints. Note that creating, updating and revoking a key require a signed-in session (JWT): an inf_ key cannot manage keys, so that a leaked key cannot mint another one or revoke yours. Listing works with either.

Create key with routing config

{
  "name": "Production",
  "model_pool": ["zai-org/GLM-4.7-Flash", "openai/gpt-oss-120b"],
  "optimize_mode": "balanced"
}

Unrecognised parameters are rejected with a 422, not ignored. A knob that is accepted is either forwarded to the vendor or refused with a reason, so a 200 always means the request you sent is the request we ran.

Inference error bodies carry both shapes: {"error": {"message", "type", "param", "code", "request_id"}} for OpenAI SDKs, and {"detail": "..."} alongside it for clients written against the older shape. Every exit uses it, including validation failures, rate limits, timeouts and oversized bodies rejected before routing; request_id repeats the X-Request-ID header so a copied error body is enough to raise a support question. typefollows OpenAI's taxonomy (invalid_request_error, authentication_error, permission_error, rate_limit_error, insufficient_quota, api_error), so SDK retry logic works unchanged. Mid-stream failures arrive as an SSE frame shaped {"object": "error", "error": {...}} followed by [DONE], with the routing disposition in code.

Drop-in replacement for the OpenAI SDK. Create an API key in the dashboard, configure your model pool, then use it:

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    api_key="inf_your_api_key",
    base_url="https://api.inferbase.ai/api/v1/inference",
)

# Smart routing: picks the best model from your key's pool
response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "Explain quantum computing in 3 sentences."}
    ],
)
print(response.choices[0].message.content)

# Or use a specific model directly
response = client.chat.completions.create(
    model="zai-org/GLM-4.7-Flash",
    messages=[
        {"role": "user", "content": "Hello!"}
    ],
)
print(response.choices[0].message.content)

Node.js (OpenAI SDK)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "inf_your_api_key",
  baseURL: "https://api.inferbase.ai/api/v1/inference",
});

const response = await client.chat.completions.create({
  model: "auto",
  messages: [
    { role: "user", content: "Explain quantum computing in 3 sentences." }
  ],
});
console.log(response.choices[0].message.content);

curl (Streaming)

curl -N -X POST https://api.inferbase.ai/api/v1/inference/chat/completions \
  -H "Authorization: Bearer inf_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'