Serverless inference with LLM routing built in
Open models through one OpenAI-compatible API, on managed capacity, billed per token. No GPUs to provision, nothing idle to pay for, and the router on whenever you send auto.
client = OpenAI(
base_url="https://api.inferbase.ai/v1",
api_key=os.environ["INFERBASE_API_KEY"],
)
stream = client.chat.completions.create(
model="deepseek-v4-flash", # or "auto"
messages=[{"role": "user", "content": prompt}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")What serverless inference is
A model called over an API on capacity someone else runs, paid per token rather than per GPU-hour, nothing left running in between.
With dedicated inference you provision GPUs, load a model, keep it warm and pay whether or not it is busy. With serverless inference the model is already loaded on a provider's hardware; you send a request, the response streams back, and the bill is the tokens. Live in minutes, nothing while idle, and the hardware, the batching and the weights are the provider's concern.
Inferbase serves the open-weight models in its catalog this way, from a managed pool of providers, behind the one API the gateway gives every application. Name a model and it is served as named; send auto and the router chooses for the task. Either way the request leaves a receipt, behind the same keys, limits and budgets.
Per token
The provider's price plus a flat 6%. No minimum, no idle charge, no capacity to reserve.
Managed pool
Open-weight models already loaded on the providers that serve them; the pool, not you, holds the hardware.
One API
OpenAI-compatible, with Anthropic-shaped clients on the Messages endpoint. Keep the SDK you have.
Routed or pinned
The router picks per request when you send auto; a named model is served as named.
How serverless inference works on Inferbase
One request in, the model resolved or chosen, a provider that already has it loaded, tokens streamed back with the record.
One OpenAI-compatible call, authenticated, checked against the key's limits and budget.
The router picks a model that clears the bar for the task, or the named model is served as named, on a provider that already has it loaded.
Streamed back through the same connection, with a usage receipt and, when routed, the decision.
One OpenAI-compatible API for serverless inference
Point an existing SDK at one base URL and keep your code. Every request shape the SDK already sends passes through.
OpenAI SDKs
Chat completions, streaming, tool calls, structured output and embeddings, in the shapes the SDK already sends.
Anthropic SDKs
Anthropic-shaped clients are served on the Messages endpoint, including streaming and the full tool loop.
Capabilities honoured
A request that carries a JSON schema or tools is served only by a model whose deployment supports them.
Frameworks
Anything that speaks the OpenAI API works unchanged: LangChain, agent frameworks, batch jobs, RAG pipelines.
The open models serverless inference serves
A curated catalog, not a logo wall: every entry carries its provenance, its verified variants, benchmark evidence where it exists, and the live price.
The managed pool serves open-weight models from 11 creators: Qwen, DeepSeek, Z.ai, Meta, Google, Moonshot AI, MiniMax, Mistral, OpenAI, NVIDIA and Microsoft. The catalog lists many more models than the pool serves, and marks which are servable today, so the model behind your API is always a known quantity: what it is, who made it, which variant is running, and what it costs.
Provenance
Each model is attributed to its creator; a variant is a variant, never passed off as the original.
Evidence
Benchmark scores per task from a public, cited source, where they exist. Where they do not, the catalog says so.
Live prices
Per-token prices read from the providers and kept current, on the catalog and on every receipt.
DeepSeek V4 Flash290.9B · 1M ctxDeepSeek
Gemma 4 31B IT31.3B · 262K ctxGoogle
- DeepSeek V4 Flash 0731304.2B · 1M ctx
DeepSeek
- Gemma 3 12B12.2B · 131K ctx
Google
Parakeet TDT 0.6B V3627.1M · 448 ctxNVIDIA
Every model listed here answered a one-token probe; the full servable set is on the catalog.
Serverless inference with LLM routing
Send auto and each request is classified for its task, narrowed to the models that clear the quality bar, and served by the one that wins on your objective.
Serverless inference is how a model runs; routing is the decision of which one. The objective is yours per key or per request: cost, balanced, quality or latency. The decision takes about a tenth of a second at the median on typical prompts, arrives before the first token when you stream, and is kept by request id. A pinned model skips the router and is served as named.
Objectives
Cost, balanced, quality, latency. Set on the key, override on the request.
Quality bar
A model has to clear the bar for the task before price is considered; unmeasured models are not assumed equal.
Session affinity
A session stays on its model across turns unless a switch is worth the cost.
Failover, budgets and warm capacity
A routed request carries a primary and two fallbacks under fixed time budgets; a pinned model falls back to another provider that serves it.
Fallback chain
A primary and up to two fallbacks per request, every attempt on the record.
Time budgets
15, 10 and 5 seconds to first token per attempt; 30 seconds for the chain.
Cooldown
A route that fails is cooled down for a minute before it is tried again.
Slow starters
Very large models scaled to zero when idle get a longer first-token budget.
Honest errors
Invalid requests and refusals are returned, not retried.
Health probes
One-token probes on every route; servable means it answered.
Serverless inference pricing: per token, no idle
Free within the free plan's limits; on Pro, the provider's token price plus a flat 6%, on every receipt.
Free plan
Chat, embeddings and rerank on our models at no charge: 100 requests and $0.50 of usage a day, 5,000 requests a month.
Tokens on Pro
The provider's per-token price plus a flat 6%, from credit (top-ups from $10). No minimum spend, no idle charge, no reserved capacity.
Plan
Pro is $19 a month with 50,000 requests on any model and any key, routed or pinned.
Caps
A requests-per-minute limit and a monthly spend cap per key, a monthly budget per project, enforced before anything is forwarded.
Frequently asked questions about serverless inference
What serverless inference is, how it differs from self-hosted GPUs and from a gateway, and how serving on Inferbase behaves.
Serverless inference is calling a model over an API on capacity that someone else runs, and paying per token rather than per hour of GPU. You send a request, the model is already loaded on the provider's hardware, the response streams back, and nothing of yours is left running between requests. The alternative is dedicated inference: GPUs you provision, load a model onto, keep warm and pay for whether or not they are busy.
Your application calls one endpoint with an API key. The gateway authenticates the key, checks its limits and budget, resolves the model you named (or, with model="auto", lets the router choose one), forwards the request to a provider in the managed pool that serves that model, streams the tokens back through the same connection, and writes a usage receipt: model, tokens in and out, cost. The provider's capacity, scaling and hardware are its concern; the API, the record and the controls are Inferbase's.
Cost shape and control. Serverless bills per token with no minimum, so idle time costs nothing, and it is live in minutes; the hardware, the batching and the model weights are the provider's. Self-hosting bills per GPU-hour whether or not the GPUs are busy, needs capacity planning and someone to run it, and in return gives you full control over the model, its quantization, its data path and its latency. Below a steady, high utilization, serverless is cheaper; above it, dedicated capacity can be.
When utilization is high and steady enough that GPU-hours cost less than the tokens they would produce serverlessly, when a policy requires the weights and the data never to leave your infrastructure, or when you need a model or a quantization no provider serves. The GPU capacity planner on this site sizes that decision: it estimates the GPUs, memory and throughput a given model and workload need, so the comparison is made on numbers rather than instinct.
Serverless inference is the execution: a model on someone's hardware answering a request. An AI gateway is the layer in front of it that gives every application one endpoint and one set of credentials, and that enforces keys, rate limits, budgets, retention and failover across every model behind it. On Inferbase they come together: the gateway is the API you call, serverless inference on the managed pool is where a request runs unless you bring your own provider keys.
Open-weight models from the curated catalog, each with a live route on the managed pool: at the time of writing, models from eleven creators including Qwen, DeepSeek, Z.ai, Meta, Google, Moonshot AI, MiniMax, Mistral, OpenAI, NVIDIA and Microsoft. Each catalog entry carries its provenance, its verified variants, benchmark evidence where it exists and the live price, so the model behind your API is a known quantity. The catalog lists many more models than the pool serves; the /models page marks which are servable today.
Yes. Point an existing OpenAI SDK at the Inferbase base URL and keep your code: chat completions, streaming, tool calls, structured output and embeddings pass through in the shapes the SDK already sends. Anthropic-shaped clients are served on the Messages endpoint, including streaming and the full tool loop. Anything that speaks the OpenAI API, from LangChain to an agent framework to a batch job, works unchanged.
All four. Streaming is served over the same server-sent events shape the OpenAI SDK expects. Tool calls and structured output (JSON mode and JSON schema) are honoured on the models whose deployments support them; when the router is choosing, a request that carries a schema or tools goes only to a model that can honour it. Embeddings have their own endpoint in the same API.
Mostly no, and where there are, they are budgeted for. The providers in the managed pool keep the models they serve loaded, and there is no per-request start-up on Inferbase's side. A few very large mixture-of-experts models are scaled to zero by their providers when idle and take longer on a cold first hit; the router knows which those are and gives them an extended time-to-first-token budget rather than failing them. Time to first token is reported on every routed response, so your own figures are yours to read.
On the free plan, chat, embeddings and rerank on Inferbase's models are free within daily limits. On Pro ($19 a month, 50,000 requests), tokens are billed at the provider's price plus a flat 6%, with no minimum spend and no idle charge, and images, speech, transcription and video are included. On your own provider keys the tokens are billed by your provider.
A routed request carries a primary and up to two fallbacks, each with its own time-to-first-token budget (15, 10 and 5 seconds) under a 30-second ceiling for the whole chain. A transient or server-side failure moves the request to the next candidate; a client error such as an invalid request is returned as it is, since another model would reject it identically. A route that fails is cooled down for a minute before it is tried again, and every attempt is written to the decision record. A pinned model falls back to another provider serving the same model where one exists.
Yes. Name a model instead of auto and it is served as named, with the same key limits, receipts and failover, and it costs the same as a routed request. Routing is a per-request choice: an application can pin where it must and send auto where it can, and switch between the two without a deploy.
Yes. Connect your own OpenAI, Anthropic, Gemini or other provider accounts to a project and the gateway serves, and if you ask, routes, on them; the credentials are encrypted at rest and the tokens are billed by your provider. Without your own keys, requests run on the managed pool. Receipts, decisions, limits and budgets apply the same way on either.
A prompt goes to the provider that serves the model chosen or named, and to nowhere else. On Inferbase, prompt and response content is kept for 30 days by default so that your own activity log and routing audit can show it, then purged; usage receipts and decision records without content are kept longer. Zero-retention can be set per user or for a whole organization, and where it changes a routing choice the decision record says so.
Both, per key, enforced at request time before anything is forwarded. Each API key carries a requests-per-minute limit and a monthly spend cap, and each project carries a monthly budget. A request over the rate limit is refused with a 429, and one that would exceed a cap or budget is refused with a 402, so a runaway loop stops at the gateway rather than on the invoice.
Put intelligence in the middle.
One gateway in front of every model, with your policies applied and every decision on record. Start on the free plan with 5,000 requests a month, no card required.