Serverless inference with LLM routing built in
Open models through one OpenAI-compatible API, on managed capacity, billed per token. No GPUs to provision, nothing idle to pay for, and the router on whenever you send auto.
client = OpenAI(
base_url="https://api.inferbase.ai/v1",
api_key=os.environ["INFERBASE_API_KEY"],
)
stream = client.chat.completions.create(
model="deepseek-v4-flash", # or "auto"
messages=[{"role": "user", "content": prompt}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")What serverless inference is
A model called over an API on capacity someone else runs, paid per token rather than per GPU-hour, nothing left running in between.
With dedicated inference you provision GPUs, load a model, keep it warm and pay whether or not it is busy. With serverless inference the model is already loaded on a host's hardware; you send a request, the response streams back, and the bill is the tokens. Live in minutes, nothing while idle, and the hardware, the batching and the weights are the host's concern.
Inferbase serves the open-weight models in its catalog this way, from a managed pool of hosts, behind the one API the gateway gives every application. Name a model and it is served as named; send auto and the router chooses for the task. Either way the request leaves a receipt, behind the same keys, limits and budgets.
Per token
The host's price plus a flat 6%. No minimum, no idle charge, no capacity to reserve.
Managed pool
Open-weight models already loaded on the hosts that serve them; the pool, not you, holds the hardware.
One API
OpenAI-compatible, with Anthropic-shaped clients on the Messages endpoint. Keep the SDK you have.
Routed or pinned
The router picks per request when you send auto; a named model is served as named.
How serverless inference works on Inferbase
One request in, the model resolved or chosen, a host that already has it loaded, tokens streamed back with the record.
One OpenAI-compatible call, authenticated, checked against the key's limits and budget.
The router picks a model that clears the bar for the task, or the named model is served as named, on a host that already has it loaded.
Streamed back through the same connection, with a usage receipt and, when routed, the decision.
One OpenAI-compatible API for serverless inference
Point an existing SDK at one base URL and keep your code. Every request shape the SDK already sends passes through.
OpenAI SDKs
Chat completions, streaming, tool calls, structured output and embeddings, in the shapes the SDK already sends.
Anthropic SDKs
Anthropic-shaped clients are served on the Messages endpoint, including streaming and the full tool loop.
Capabilities honoured
A request that carries a JSON schema or tools is served only by a model whose deployment supports them.
Frameworks
Anything that speaks the OpenAI API works unchanged: LangChain, agent frameworks, batch jobs, RAG pipelines.
The open models serverless inference serves
A curated catalog, not a logo wall: every entry carries its provenance, its verified variants, benchmark evidence where it exists, and the live price.
The managed pool serves open-weight models from 11 creators: Qwen, DeepSeek, Z.ai, Meta, Google, Moonshot AI, MiniMax, Mistral, OpenAI, NVIDIA and Microsoft. The catalog lists many more models than the pool serves, and marks which are servable today, so the model behind your API is always a known quantity: what it is, who made it, which variant is running, and what it costs.
Provenance
Each model is attributed to its creator; a variant is a variant, never passed off as the original.
Evidence
Benchmark scores per task from a public, cited source, where they exist. Where they do not, the catalog says so.
Live prices
Per-token prices read from the hosts and kept current, on the catalog and on every receipt.
Gemma 4 31B IT31.3B · 262K ctxGoogle
235B · 131K ctx- Gemma 3 12B12.2B · 131K ctx
Google
- Qwen 3.5 9B9.7B · 262K ctx
Qwen
Minimax M2.7228.7B · 205K ctxMinimax
Every model listed here answered a one-token probe; the full servable set is on the catalog.
Serverless inference with LLM routing
Send auto and each request is classified for its task, narrowed to the models that clear the quality bar, and served by the one that wins on your objective.
Serverless inference is how a model runs; routing is the decision of which one. The objective is yours per key or per request: cost, balanced, quality or latency. The decision takes about a tenth of a second at the median on typical prompts, arrives before the first token when you stream, and is kept by request id. A pinned model skips the router and is never charged a routing fee.
Objectives
Cost, balanced, quality, latency. Set on the key, override on the request.
Quality bar
A model has to clear the bar for the task before price is considered; unmeasured models are not assumed equal.
Session affinity
A conversation stays on its model across turns unless a switch is worth the cost.
Failover, budgets and warm capacity
A routed request carries a primary and two fallbacks under fixed time budgets; a pinned model falls back to another host that serves it.
Fallback chain
A primary and up to two fallbacks per request, every attempt on the record.
Time budgets
15, 10 and 5 seconds to first token per attempt; 30 seconds for the chain.
Cooldown
A route that fails is cooled down for a minute before it is tried again.
Slow starters
Very large models scaled to zero when idle get a longer first-token budget.
Honest errors
Invalid requests and refusals are returned, not retried.
Health probes
One-token probes on every route; servable means it answered.
A transient failure moves to the next model in the chain; an invalid request or a content refusal is returned as it is, because another model would reject it identically.
Serverless inference pricing: per token, no idle
Two meters, both visible on every receipt. Free during beta; the published plan is pay as you go.
Tokens
The host's per-token price plus a flat 6%. No minimum spend, no idle charge, no reserved capacity.
Decisions
$1 per 1,000 routing decisions, the first 5,000 each month free. A pinned model is never charged one.
Credit
New accounts start with $5 of credit. Top-ups from $10.
Caps
A requests-per-minute limit and a monthly spend cap per key, a monthly budget per project, enforced before anything is forwarded.
Serve on the managed pool, or on your own accounts
Connect your own provider accounts and the gateway serves and routes on them; without them, requests run on the managed pool.
Your accounts
OpenAI, Anthropic, Gemini and other provider keys, connected per project and encrypted at rest.
Your bill
Tokens are billed by your provider; the only Inferbase charge is the routing decision.
Managed pool
Without keys, requests run on the curated catalog at the host's token price plus a flat 6%.
Your key
served on your OpenAI, Anthropic or Gemini account, connected per project
tokens billed by your provider · Inferbase charges the routing decision
Managed catalog
served on the curated catalog, no keys to connect
tokens at the provider price plus a flat 6%
Same record
- receipt
- decision
- limits
- budgets
How serverless inference on Inferbase compares
Aggregators resell hosts with no decision, proxies follow your rules, routers only recommend. Inferbase decides, serves and keeps the record.
| Inferbase | OpenRouter | LiteLLM | Portkey | NotDiamond | RouteLLM | |
|---|---|---|---|---|---|---|
| Picks the best model per request | First-party | Via NotDiamond add-on | Beta tiers you map by hand | No, rules you define | Yes | Strong vs weak only |
| Routes and serves in one API | Yes | Yes | Proxies via your providers | Proxies via your providers | No, you run it | No, self-hosted |
| Per-request decision audit | Yes | No | Logs and cost tracking | Deep logs and traces | Recommend-side | Build your own |
| Nothing to self-host or calibrate | Yes | Yes | No | Hosted, rules are yours | Yes | No |
| Model breadth | Curated catalog of open models | Hundreds of models | 100+ providers, your keys | 1,600+ models, your keys | Your chosen pool | Two models |
Frequently asked questions about serverless inference
What serverless inference is, how it differs from self-hosted GPUs and from a gateway, and how serving on Inferbase behaves.
Serverless inference is calling a model over an API on capacity that someone else runs, and paying per token rather than per hour of GPU. You send a request, the model is already loaded on the host's hardware, the response streams back, and nothing of yours is left running between requests. The alternative is dedicated inference: GPUs you provision, load a model onto, keep warm and pay for whether or not they are busy.
Your application calls one endpoint with an API key. The gateway authenticates the key, checks its limits and budget, resolves the model you named (or, with model="auto", lets the router choose one), forwards the request to a host in the managed pool that serves that model, streams the tokens back through the same connection, and writes a usage receipt: model, tokens in and out, cost. The host's capacity, scaling and hardware are its concern; the API, the record and the controls are Inferbase's.
Cost shape and control. Serverless bills per token with no minimum, so idle time costs nothing, and it is live in minutes; the hardware, the batching and the model weights are the host's. Self-hosting bills per GPU-hour whether or not the GPUs are busy, needs capacity planning and someone to run it, and in return gives you full control over the model, its quantization, its data path and its latency. Below a steady, high utilization, serverless is cheaper; above it, dedicated capacity can be.
When utilization is high and steady enough that GPU-hours cost less than the tokens they would produce serverlessly, when a policy requires the weights and the data never to leave your infrastructure, or when you need a model or a quantization no host serves. The GPU capacity planner on this site sizes that decision: it estimates the GPUs, memory and throughput a given model and workload need, so the comparison is made on numbers rather than instinct.
Serverless inference is the execution: a model on someone's hardware answering a request. An AI gateway is the layer in front of it that gives every application one endpoint and one set of credentials, and that enforces keys, rate limits, budgets, retention and failover across every model behind it. On Inferbase they come together: the gateway is the API you call, serverless inference on the managed pool is where a request runs unless you bring your own provider keys.
Open-weight models from the curated catalog, each with a live route on the managed pool: at the time of writing, models from eleven creators including Qwen, DeepSeek, Z.ai, Meta, Google, Moonshot AI, MiniMax, Mistral, OpenAI, NVIDIA and Microsoft. Each catalog entry carries its provenance, its verified variants, benchmark evidence where it exists and the live price, so the model behind your API is a known quantity. The catalog lists many more models than the pool serves; the /models page marks which are servable today.
Yes. Point an existing OpenAI SDK at the Inferbase base URL and keep your code: chat completions, streaming, tool calls, structured output and embeddings pass through in the shapes the SDK already sends. Anthropic-shaped clients are served on the Messages endpoint, including streaming and the full tool loop. Anything that speaks the OpenAI API, from LangChain to an agent framework to a batch job, works unchanged.
All four. Streaming is served over the same server-sent events shape the OpenAI SDK expects. Tool calls and structured output (JSON mode and JSON schema) are honoured on the models whose deployments support them; when the router is choosing, a request that carries a schema or tools goes only to a model that can honour it. Embeddings have their own endpoint in the same API.
Mostly no, and where there are, they are budgeted for. The hosts in the managed pool keep the models they serve loaded, and there is no per-request start-up on Inferbase's side. A few very large mixture-of-experts models are scaled to zero by their hosts when idle and take longer on a cold first hit; the router knows which those are and gives them an extended time-to-first-token budget rather than failing them. Time to first token is reported on every routed response, so your own figures are yours to read.
Per token, at the host's price plus a flat 6%, with no minimum spend and no idle charge. Routing decisions are metered separately at $1 per 1,000, with the first 5,000 each month free; a request that names a model is never charged a routing fee. New accounts start with $5 of credit, and the service is free during beta. On your own provider keys the tokens are billed by your provider.
A routed request carries a primary and up to two fallbacks, each with its own time-to-first-token budget (15, 10 and 5 seconds) under a 30-second ceiling for the whole chain. A transient or server-side failure moves the request to the next candidate; a client error such as an invalid request is returned as it is, since another model would reject it identically. A route that fails is cooled down for a minute before it is tried again, and every attempt is written to the decision record. A pinned model falls back to another host serving the same model where one exists.
Yes. Name a model instead of auto and it is served as named, with the same key limits, receipts and failover and no routing fee. Routing is a per-request choice: an application can pin where it must and send auto where it can, and switch between the two without a deploy.
Yes. Connect your own OpenAI, Anthropic, Gemini or other provider accounts to a project and the gateway serves, and if you ask, routes, on them; the credentials are encrypted at rest and the tokens are billed by your provider. Without your own keys, requests run on the managed pool. Receipts, decisions, limits and budgets apply the same way on either.
A prompt goes to the host that serves the model chosen or named, and to nowhere else. On Inferbase, prompt and response content is kept for 30 days by default so that your own activity log and routing audit can show it, then purged; usage receipts and decision records without content are kept longer. Zero-retention can be set per user or for a whole organization, and where it changes a routing choice the decision record says so.
Both, per key, enforced at request time before anything is forwarded. Each API key carries a requests-per-minute limit and a monthly spend cap, and each project carries a monthly budget. A request over the rate limit is refused with a 429, and one that would exceed a cap or budget is refused with a 402, so a runaway loop stops at the gateway rather than on the invoice.
Put intelligence in the middle.
One gateway in front of every model, with your policies applied and every decision on record. Start with $5 of credit and 5,000 routing decisions a month, no card required.