The AI gateway with LLM routing built in
One OpenAI-compatible endpoint in front of every model you use: the managed catalog, your own provider keys, and your Azure, Bedrock or Vertex accounts. Each request is routed to the best model for the task, served, and returned with its decision on the record.
Every stage is enforced before the next; the receipt and the decision resolve from the request id on the response, in the API and the dashboard.
What an AI gateway does
Between your applications and the models they call: where an organization sets who calls what, spends how much, and what is recorded.
An AI gateway gives every service one endpoint and one set of credentials in front of many models and providers, and it handles once the concerns that would otherwise be repeated in each service: authentication, rate limits, budgets, logging and failover. Applications keep the API they already use; the gateway holds the provider accounts, the catalog and the record.
Inferbase is that layer with one addition. Where a gateway alone sends each request to the model you named, this one can also decide which model should answer, on the task and the objective you set, and it keeps the reasoning with the request. The gateway is the front of a control plane with four capabilities behind it: the catalog, routing, security and observability.
Inferbase Gateway
One OpenAI-compatible API, with failover, rate limits and budgets behind it.
- knowledge
Catalog
Curated creators, verified variants, benchmark evidence where it exists, live prices.
- decision
Routing
A prompt classified, a model chosen on your objective, the decision kept.
- policy
Guardrails
Which models may serve, guardrails at three doors, which tools an agent may call.
- record
Observability
Decision, verdicts, tokens, cost and latency per request; usage and an audit log.
One OpenAI-compatible API in front of every model
Point an existing SDK at one base URL and keep your code. Name a model to pin it, or send auto and let the gateway choose.
OpenAI SDKs
Chat completions, streaming, tool calls, structured output and embeddings pass through in the shapes the SDK already sends.
Anthropic SDKs
Anthropic-shaped clients are served on the Messages endpoint, including streaming and the full tool loop, and route the same way.
Pin or route
model="auto" lets the gateway choose; a named model is served as named. Switch per request.
Frameworks
Anything that speaks the OpenAI API works unchanged: LangChain, agent frameworks, batch jobs, RAG pipelines.
LLM routing inside the AI gateway
Set the model to auto: each request is classified for its task, narrowed to the models that clear the quality bar, and picked on your objective.
The objective is yours per key or per request: cost, balanced, quality or latency. The decision takes about a tenth of a second at the median on typical prompts, arrives before the first token when you stream, and is kept by request id with the task, the complexity, the candidates that qualified, the one that won and how clearly. Multi-turn work stays on the model that already holds the session unless switching is worth it.
Quality bar
A model has to clear the bar for the task before price is considered; unmeasured models are not assumed equal.
Session affinity
A session stays on its model across turns unless a switch is worth the cost.
AI gateway governance: keys, limits and budgets
Every control is enforced at request time, before anything is forwarded, so a runaway loop stops at the gateway rather than on the invoice.
Per-key rate limits
Requests per minute per API key; a request over the limit is refused with a 429.
Spend caps
A monthly cap per key; a request that would exceed it is refused with a 402.
Project budgets
A monthly budget per project, enforced the same way across every key in it.
Roles
Owner, admin, billing, member and viewer, gating what a member can change and see.
Multi-factor authentication
An organization can require MFA for access to its settings and control plane.
Zero retention
Per user or for a whole organization; disclosed on the decision wherever it changes a choice.
Failover, cooldown and cache-aware serving
A routed request carries a primary and two fallbacks; the gateway moves on transient failures and returns client errors as they are.
Fallback chain
Primary plus up to two fallbacks per request, each attempt written to the decision record.
Cooldown
A route that fails is cooled down for a minute before it is tried again.
Honest errors
Invalid requests and content refusals are returned, not retried; another model would reject them identically.
Cache-aware
A prompt marked for caching prefers a route that can cache it, within the quality band.
A transient failure moves to the next model in the chain; an invalid request or a content refusal is returned as it is, because another model would reject it identically.
Every request on the record
A receipt for every request and a decision for every routed one, retrievable by request id from the API and the dashboard.
Usage receipt
Model, tokens in and out, cached tokens where the provider reported them, and cost.
Routing decision
Task, complexity, objective, how many models qualified, which won and how clearly.
By request id
Both records resolve from the id on the response, in the API and in the activity log.
Streamed first
When streaming, the decision is the first event, before any content token.
Bring your own keys, or serve on ours
Connect your own provider accounts and the gateway routes and serves on them; without them, requests run on the managed catalog.
Your accounts
OpenAI, Anthropic, Gemini and other provider keys, connected per project and encrypted at rest.
Your bill
Tokens are billed by your provider; Inferbase adds nothing to them, and the request counts toward your plan.
Managed catalog
Without keys, requests run on the curated catalog at the provider token price plus a flat 6%.
Your key
served on your OpenAI, Anthropic or Gemini account, connected per project
tokens billed by your provider · nothing added by Inferbase
Managed catalog
served on the curated catalog, no keys to connect
tokens at the provider price plus a flat 6%
Same record
- receipt
- decision
- limits
- budgets
Frequently asked questions about AI gateways
What an AI gateway is, how it differs from an API gateway and a router, and how this one behaves.
An AI gateway is the layer that sits between your applications and the AI models they call. It gives every service one endpoint and one set of credentials in front of many models and providers, and it handles the concerns that would otherwise be repeated in each service: authentication, rate limits, budgets, logging, and failover. An organization uses it to decide who may call which model, how much they may spend, and what gets recorded.
Your application sends a request to the gateway instead of to a provider. The gateway authenticates the key, checks its limits and budget, chooses or confirms the model, forwards the request to the provider that serves it, streams the response back, and writes a record of what happened. To the application it looks like one API; behind it the gateway can move between models and providers without a code change on your side.
An API gateway routes HTTP traffic to your own services and knows nothing about what is inside a request. An AI gateway understands the request as a model call: it can pick a model, count tokens, price the call, apply a budget in dollars rather than requests, keep the prompt out of its logs when a policy says so, and fall back to a different model when one fails. The two can coexist; the AI gateway is the one that understands models.
An LLM router decides which model should answer a request; an AI gateway executes the request and governs it. Some routers only recommend, leaving you to call the provider yourself, and some gateways only proxy, sending every request to the model you named. Inferbase does both in one API: the router decides inside the gateway, and the gateway serves the result and keeps the decision on record.
Between the application code (or agent framework) and the model providers. Your services talk to the gateway through the OpenAI-compatible API they already use; the gateway holds the provider credentials, the model catalog, the limits, and the logs. Nothing else in the stack needs to know which provider served a given request, which is what makes the models interchangeable.
Yes. Point an existing OpenAI SDK at the Inferbase base URL and keep your code: chat completions, streaming, tool calls, structured output, and embeddings pass through in the shapes the SDK already sends. Anthropic-shaped clients are served on the Messages endpoint, so an Anthropic SDK routes the same way.
Either. Send model="auto" and the gateway classifies the request and chooses the model on your objective (cost, balanced, quality, or latency). Name a model instead and the gateway serves that model, with the same limits, receipts, and failover. You can switch per request, so an application can pin where it must and route where it can.
For a pinned model, the gateway adds the overhead of authentication, limits, and logging, which is small next to model latency. For a routed request, classification and the decision take about a tenth of a second at the median on typical prompts, and more on very long ones (measured in July 2026). The time spent is reported on every routed response as routing_time_ms, so you can read your own figure rather than ours.
Every request writes a usage receipt: the model, tokens in and out, cached tokens where the provider reported them, and cost. Routed requests also keep their decision: the task the classifier saw, its complexity, the objective, how many models qualified, which one won and how clearly. Both are retrievable by request id from the API and in the dashboard.
At request time, before anything is forwarded. Each API key carries a requests-per-minute limit and a monthly spend cap; each project carries a monthly budget. A request over the rate limit is refused with a 429, and a request that would exceed a cap or budget is refused with a 402, so a runaway loop stops at the gateway rather than on your invoice.
Yes. Connect your own OpenAI, Anthropic, Gemini, or other provider accounts to a project and the gateway routes and serves on them. The credentials are encrypted at rest, the tokens are billed by your provider, and Inferbase adds nothing to them; the request counts toward your plan like any other. Without your own keys, requests run on the managed catalog and are billed per token at the provider price plus a flat 6%.
A routed request carries a primary and up to two fallbacks. If an attempt fails with a transient or server-side error, the gateway moves to the next candidate; client errors such as an invalid request are returned as they are, since another model would reject them identically. A route that fails is cooled down for a minute before it is tried again, and every attempt is written to the decision record.
By default, prompt and response content is kept for 30 days so that your own activity log and routing audit can show it, then purged. Zero-retention can be set per user or for a whole organization; with it on, content is not stored, and wherever that changes a routing choice (for example, prompt caching), the decision record says so.
Yes. An organization holds projects, and each project holds its keys, connections, and budget. Members carry roles (owner, admin, billing, member, viewer) that gate what they can change and see, and an organization can require multi-factor authentication for access to its settings. Usage and decisions are visible to those the roles allow and to no one else.
A free plan with no time limit and no card: 5,000 requests a month on any model and any key, with Inferbase's own models free for chat, embeddings and rerank within daily limits. Pro is $19 a month with 50,000 requests and no daily limits. Managed inference on Pro is billed at the provider's token price plus a flat 6%; on your own provider keys or cloud account the tokens are billed by your provider, with nothing added. Routed or pinned, a request counts the same.
Put intelligence in the middle.
One gateway in front of every model, with your policies applied and every decision on record. Start on the free plan with 5,000 requests a month, no card required.