The AI gateway with LLM routing built in
One OpenAI-compatible endpoint in front of a curated model catalog. Each request is routed to the best model for the task, served, and returned with its decision on the record.
Every stage is enforced before the next; the receipt and the decision resolve from the request id on the response, in the API and the dashboard.
What an AI gateway does
Between your applications and the models they call: where an organization sets who calls what, spends how much, and what is recorded.
An AI gateway gives every service one endpoint and one set of credentials in front of many models and providers, and it handles once the concerns that would otherwise be repeated in each service: authentication, rate limits, budgets, logging and failover. Applications keep the API they already use; the gateway holds the provider accounts, the catalog and the record.
Inferbase is that layer with one addition. Where a gateway alone sends each request to the model you named, this one can also decide which model should answer, on the task and the objective you set, and it keeps the reasoning with the request. The gateway is the execution layer of a control plane whose other two layers are the router and the catalog.
AI Gateway
One API, receipts, limits, budgets, failover. The execution layer.
LLM Router
A prompt classified, a model chosen on your objective, the decision kept. The decision layer.
Model Catalog
Curated creators, verified variants, benchmark evidence where it exists, live prices. The knowledge layer.
One OpenAI-compatible API in front of the catalog
Point an existing SDK at one base URL and keep your code. Name a model to pin it, or send auto and let the gateway choose.
OpenAI SDKs
Chat completions, streaming, tool calls, structured output and embeddings pass through in the shapes the SDK already sends.
Anthropic SDKs
Anthropic-shaped clients are served on the Messages endpoint, including streaming and the full tool loop, and route the same way.
Pin or route
model="auto" lets the gateway choose; a named model is served as named. Switch per request.
Frameworks
Anything that speaks the OpenAI API works unchanged: LangChain, agent frameworks, batch jobs, RAG pipelines.
LLM routing inside the AI gateway
Set the model to auto: each request is classified for its task, narrowed to the models that clear the quality bar, and picked on your objective.
The objective is yours per key or per request: cost, balanced, quality or latency. The decision takes about a tenth of a second at the median on typical prompts, arrives before the first token when you stream, and is kept by request id with the task, the complexity, the candidates that qualified, the one that won and how clearly. Multi-turn work stays on the model that already holds the conversation unless switching is worth it.
Objectives
Cost, balanced, quality, latency. Set on the key, override on the request.
Quality bar
A model has to clear the bar for the task before price is considered; unmeasured models are not assumed equal.
Session affinity
A conversation stays on its model across turns unless a switch is worth the cost.
AI gateway governance: keys, limits and budgets
Every control is enforced at request time, before anything is forwarded, so a runaway loop stops at the gateway rather than on the invoice.
Per-key rate limits
Requests per minute per API key; a request over the limit is refused with a 429.
Spend caps
A monthly cap per key; a request that would exceed it is refused with a 402.
Project budgets
A monthly budget per project, enforced the same way across every key in it.
Roles
Owner, admin, billing, member and viewer, gating what a member can change and see.
Multi-factor authentication
An organization can require MFA for access to its settings and control plane.
Zero retention
Per user or for a whole organization; disclosed on the decision wherever it changes a choice.
Failover, cooldown and cache-aware serving
A routed request carries a primary and two fallbacks; the gateway moves on transient failures and returns client errors as they are.
Fallback chain
Primary plus up to two fallbacks per request, each attempt written to the decision record.
Cooldown
A route that fails is cooled down for a minute before it is tried again.
Honest errors
Invalid requests and content refusals are returned, not retried; another model would reject them identically.
Cache-aware
A prompt marked for caching prefers a route that can cache it, within the quality band.
Every request on the record
A receipt for every request and a decision for every routed one, retrievable by request id from the API and the dashboard.
Usage receipt
Model, tokens in and out, cached tokens where the provider reported them, and cost.
Routing decision
Task, complexity, objective, how many models qualified, which won and how clearly.
By request id
Both records resolve from the id on the response, in the API and in the activity log.
Streamed first
When streaming, the decision is the first event, before any content token.
Bring your own keys, or serve on ours
Connect your own provider accounts and the gateway routes and serves on them; without them, requests run on the managed catalog.
Your accounts
OpenAI, Anthropic, Gemini and other provider keys, connected per project and encrypted at rest.
Your bill
Tokens are billed by your provider; the only Inferbase charge is the routing decision.
Managed catalog
Without keys, requests run on the curated catalog at the provider token price plus a flat 6%.
Same record
Receipts, decisions, limits and budgets apply the same way on either.
How this AI gateway compares
Routers recommend, aggregators resell routing, frameworks make you host it. Inferbase decides, serves and keeps the record, in one API.
| Inferbase | OpenRouter | LiteLLM | Portkey | NotDiamond | RouteLLM | |
|---|---|---|---|---|---|---|
| Picks the best model per request | First-party | Via NotDiamond add-on | Beta tiers you map by hand | No, rules you define | Yes | Strong vs weak only |
| Routes and serves in one API | Yes | Yes | Proxies via your providers | Proxies via your providers | No, you run it | No, self-hosted |
| Per-request decision audit | Yes | No | Logs and cost tracking | Deep logs and traces | Recommend-side | Build your own |
| Nothing to self-host or calibrate | Yes | Yes | No | Hosted, rules are yours | Yes | No |
| Model breadth | Curated catalog of open models | Hundreds of models | 100+ providers, your keys | 1,600+ models, your keys | Your chosen pool | Two models |
Frequently asked questions about AI gateways
What an AI gateway is, how it differs from an API gateway and a router, and how this one behaves.
An AI gateway is the layer that sits between your applications and the AI models they call. It gives every service one endpoint and one set of credentials in front of many models and providers, and it handles the concerns that would otherwise be repeated in each service: authentication, rate limits, budgets, logging, and failover. An organization uses it to decide who may call which model, how much they may spend, and what gets recorded.
Your application sends a request to the gateway instead of to a provider. The gateway authenticates the key, checks its limits and budget, chooses or confirms the model, forwards the request to the provider that serves it, streams the response back, and writes a record of what happened. To the application it looks like one API; behind it the gateway can move between models and providers without a code change on your side.
An API gateway routes HTTP traffic to your own services and knows nothing about what is inside a request. An AI gateway understands the request as a model call: it can pick a model, count tokens, price the call, apply a budget in dollars rather than requests, keep the prompt out of its logs when a policy says so, and fall back to a different model when one fails. The two can coexist; the AI gateway is the one that understands models.
An LLM router decides which model should answer a request; an AI gateway executes the request and governs it. Some routers only recommend, leaving you to call the provider yourself, and some gateways only proxy, sending every request to the model you named. Inferbase does both in one API: the router decides inside the gateway, and the gateway serves the result and keeps the decision on record.
Between the application code (or agent framework) and the model providers. Your services talk to the gateway through the OpenAI-compatible API they already use; the gateway holds the provider credentials, the model catalog, the limits, and the logs. Nothing else in the stack needs to know which provider served a given request, which is what makes the models interchangeable.
Yes. Point an existing OpenAI SDK at the Inferbase base URL and keep your code: chat completions, streaming, tool calls, structured output, and embeddings pass through in the shapes the SDK already sends. Anthropic-shaped clients are served on the Messages endpoint, so an Anthropic SDK routes the same way.
Either. Send model="auto" and the gateway classifies the request and chooses the model on your objective (cost, balanced, quality, or latency). Name a model instead and the gateway serves that model, with the same limits, receipts, and failover. You can switch per request, so an application can pin where it must and route where it can.
For a pinned model, the gateway adds the overhead of authentication, limits, and logging, which is small next to model latency. For a routed request, classification and the decision take about a tenth of a second at the median on typical prompts, and more on very long ones (measured in July 2026). The time spent is reported on every routed response as routing_time_ms, so you can read your own figure rather than ours.
Every request writes a usage receipt: the model, tokens in and out, cached tokens where the provider reported them, and cost. Routed requests also keep their decision: the task the classifier saw, its complexity, the objective, how many models qualified, which one won and how clearly. Both are retrievable by request id from the API and in the dashboard.
At request time, before anything is forwarded. Each API key carries a requests-per-minute limit and a monthly spend cap; each project carries a monthly budget. A request over the rate limit is refused with a 429, and a request that would exceed a cap or budget is refused with a 402, so a runaway loop stops at the gateway rather than on your invoice.
Yes. Connect your own OpenAI, Anthropic, Gemini, or other provider accounts to a project and the gateway routes and serves on them. The credentials are encrypted at rest, the tokens are billed by your provider, and the only Inferbase charge is the routing decision. Without your own keys, requests run on the managed catalog and are billed per token.
A routed request carries a primary and up to two fallbacks. If an attempt fails with a transient or server-side error, the gateway moves to the next candidate; client errors such as an invalid request are returned as they are, since another model would reject them identically. A route that fails is cooled down for a minute before it is tried again, and every attempt is written to the decision record.
By default, prompt and response content is kept for 30 days so that your own activity log and routing audit can show it, then purged. Zero-retention can be set per user or for a whole organization; with it on, content is not stored, and wherever that changes a routing choice (for example, prompt caching), the decision record says so.
Yes. An organization holds projects, and each project holds its keys, connections, and budget. Members carry roles (owner, admin, billing, member, viewer) that gate what they can change and see, and an organization can require multi-factor authentication for access to its settings. Usage and decisions are visible to those the roles allow and to no one else.
It is free during beta. The published plan is pay as you go with two meters: smart-routing decisions at $1 per 1,000 with the first 5,000 each month free (a pinned model is never charged a routing fee), and managed inference at the provider token price plus a flat 6%. On your own provider keys the tokens are billed by your provider. New accounts start with $5 of credit.
Route every request to the right model.
One OpenAI-compatible endpoint, the best model for each request, every decision on the record. New models and prices reach you with no rewrite.