The vocabulary around LLM infrastructure has settled faster than the understanding of it. Teams shopping for a way out of single-provider lock-in encounter gateways, routers, aggregators, and proxies, often described in each other's terms, and the two words that get used interchangeably most often are gateway and router. The confusion is understandable: both sit between your application and the models, both expose one OpenAI-compatible endpoint, and both promise that requests end up in the right place.
The distinction matters more than the naming suggests, because the two layers decide different things. A gateway decides how a request reaches the model you already chose. A router decides which model the request should reach in the first place. A team that adopts a gateway believing it bought model selection ends up maintaining that selection by hand in a config file, and a team that only needed unified key management does not need a routing brain. This post separates the two layers, gives a concrete test for telling them apart, and covers when each one, or both, is the right tool. It builds on our guide to model routing, which covers the routing layer itself in more depth.
What an LLM gateway does
An LLM gateway is a unification layer over many model providers. You put it in front of the provider accounts you hold, and it gives every application one API surface, one place to manage credentials, and one stream of telemetry. The strongest gateways are genuinely good at this: LiteLLM is an open-source proxy that fronts more than a hundred providers with cost tracking, budgets, and guardrails, and Portkey layers deep observability, prompt management, and a guardrail library over a similar surface. For platform teams supporting many internal consumers of LLM APIs, this control plane is the product.
Gateways also route, in a specific sense of the word. LiteLLM's routing strategies balance load across deployments of the model you named, picking a host by latency, cost, or availability, and its fallbacks walk an ordered list you define when a call fails. Its beta AutoRouter goes a step further, classifying prompts into complexity tiers by heuristics, keyword rules, or an optional small-model call, though which model serves each tier remains a mapping you write. Portkey's conditional routing matches rules against metadata, request parameters, or the URL path, and sends the request to whichever target the first matching rule names. Retries, circuit breakers, and canary splits round out the toolkit.
Notice what every one of those mechanisms has in common: the options were written down by a person before the request existed. The gateway executes reliability and governance policy. It does not form an opinion about the request.
What an LLM router decides
An LLM router makes the model choice itself, for each request individually. Instead of executing a rule that names a model, it inspects the prompt, estimates what the request needs, and scores the candidate models against that estimate on the objective you set, whether that is quality, cost, or latency. The estimate typically comes from a lightweight classifier that predicts the task family and difficulty; the scoring draws on per-model evidence such as benchmark results, prices, and observed latency. We describe this mechanism in detail in What Is Model Routing.
The reason to want this is the shape of production traffic. Request streams are heterogeneous: a support workload mixes password resets with genuinely hard multi-step problems, and a coding assistant mixes docstring requests with cross-file refactors. Research on the economics of this mix, from FrugalGPT through RouteLLM, has repeatedly found that a large share of real traffic can be served by smaller models with little or no quality loss, with savings in the range of 50 to 80 percent depending on the workload and the quality bar. A fixed model choice prices every request at the rate of the hardest one; a router prices each request individually.
A production router carries reliability machinery too, and this is where the layers genuinely overlap. When the chosen model's provider fails or times out, the router fails over within the candidate pool, the same job a gateway's fallback list performs. The difference is how the fallback order came to exist: in a gateway it is a list someone wrote and must now maintain; in a router it is ranked from the same evidence that drove the original choice. We cover the failure-handling side of this in LLM fallback chains.
The test that separates them
Marketing pages blur these categories, so it helps to have a test that does not depend on what a product calls itself: can the layer change the model based on the prompt content alone?
Ask what inputs are allowed to influence the model decision. In a gateway, every input a rule can key on exists before the prompt is read: the config file, metadata your application attached, the calling key, the error and rate-limit state of providers. Nothing in the rule engine reads the request body and asks whether this particular prompt would be better served by a different model. In a router, the prompt is the primary input, and the decision can differ between two requests that are byte-for-byte identical in their metadata.
Applying the test sorts the market quickly, and it also handles the boundary cases the marketing labels blur. LiteLLM's deployment strategies and hand-written fallback lists sit on the gateway side, as do Portkey's metadata conditions. LiteLLM's beta AutoRouter is the instructive middle case: it does read the prompt, but what it computes is a coarse complexity tier, and the model behind each tier is still a line in your config rather than a prediction about which model would answer this prompt best. Reading the prompt to bucket it is not the same as estimating per-model quality for it. OpenRouter's default routing is provider-level, choosing the cheapest reliable host for the model you named, though its optional Auto Router does cross into model selection by delegating the choice to NotDiamond. RouteLLM and NotDiamond are routers by this test, with the caveat that they only recommend or relay; running the chosen model remains your problem. A routing-first platform passes the test and serves the result.
Side by side
| Dimension | LLM gateway | LLM router |
|---|---|---|
| Core decision | How the request reaches the model you named | Which model the request should reach |
| Decision input | Config, metadata, provider state | The prompt itself, plus model evidence |
| Who writes the logic | You, as rules and fallback lists | The platform, from per-task evidence |
| Optimizes | Reliability, governance, provider cost for a fixed model | Quality, cost, and latency per request |
| Failure handling | Ordered fallback lists you define | Ranked failover within the candidate pool |
| Maintenance | Config tracks the model landscape by hand | Evidence updates absorb model churn |
| Typical buyer | Platform team governing many internal consumers | Product team with mixed traffic and a bill |
The maintenance row deserves emphasis, because it is the cost that shows up months later. A gateway config encodes yesterday's model landscape: every new model release, price change, and deprecation is an edit someone owns, and the fallback list that was sensible in January quietly stops being sensible by June. Providers change prices without announcements more often than teams expect, a pattern we have measured across our own catalog and cover in the hidden costs of LLM APIs. Rules do not notice any of this. Evidence does.
When a gateway is the right tool
If the problem is governance, a gateway solves it and a router does not. An organization with a dozen teams calling LLM APIs on scattered credentials needs shared key management, per-team budgets, audit logs, and guardrails far more than it needs per-request model selection, and it needs them inside its own network boundary in regulated settings. Self-hosted gateways fit compliance requirements that no hosted service can, and a gateway is also the honest choice when model selection is genuinely a non-problem: one provider, one model, uniform traffic.
The trade is operational. You run the proxy, hold the provider accounts, and own the config as the model landscape shifts underneath it. For a platform team whose job is exactly that, the trade is fine, and often correct.
When a router is the right tool
If the problem is that your traffic is mixed and your bill or your quality is suffering for it, that is router territory. The symptoms are recognizable: a frontier-model bill dominated by requests that did not need one, or a cheaper default model that quietly underperforms on the hard minority. A rules layer cannot fix this, because the information that distinguishes an easy request from a hard one lives in the prompt, which rules cannot see.
The same applies if you do not want to hold provider accounts or maintain a selection config at all. A routing-first platform makes the model decision from the request, serves the chosen model, and keeps a per-request record of what was decided and why. Our own comparisons with LiteLLM and with Portkey go dimension by dimension on where each fits, including the cases where the gateway is the better choice.
A worked example
Numbers make the difference concrete. Take a support assistant handling 100,000 requests a month, averaging 1,500 input and 400 output tokens per request. Suppose 75 percent of the traffic is routine (password resets, order status, short factual answers), 20 percent is moderate (multi-step troubleshooting), and 5 percent is genuinely hard (ambiguous, multi-system problems).
Served entirely by a frontier-class model at a blended rate of roughly $6 per million tokens, the month costs about $1,140, and every password reset paid the frontier rate. Behind a gateway, the picture improves only where you hand-optimize it: the gateway can balance hosts for that one model and fail over when a provider degrades, but every request still goes to the model the config names, at that model's price. The bill moves only if someone edits the config, and someone has to notice that it should move.
A router prices the tiers separately because it can see which tier each request belongs to. If the routine 75 percent lands on small models at roughly $0.30 per million tokens and the moderate 20 percent on mid-tier models at roughly $1.50, with the hard 5 percent still going to the frontier model, the same month costs around $210, an 80 percent reduction with the hard cases served exactly as before. The specific rates shift as the market moves, and the tier mix is workload-dependent, so treat these as illustrative rather than a quote. The structural point survives any particular set of prices: a fixed model choice prices the whole distribution at the rate of its tail, and no amount of gateway configuration changes that, because the config cannot see the distribution.
Running both together
The categories compose, because they solve different problems at the same point in the stack. A platform team can keep its gateway as the organization-wide control point for keys, budgets, and guardrails, and register a router behind it as one more OpenAI-compatible endpoint. Teams whose traffic benefits from per-request model selection point at that endpoint; teams that pinned a model for good reasons keep their pin. Governance stays where the organization put it, and the model decision moves to the layer that can actually see the request.
What does not compose well is expecting one layer to do the other's job. A gateway asked to do model selection becomes a growing rules file that approximates a router badly and ages worse. A router asked to be an organization's governance layer is missing the point in the other direction: its job is the decision, not your org chart.
What this means in practice
The gateway question and the router question are different questions, and answering the wrong one is how teams end up disappointed with tools that work exactly as documented. Ask what is actually hurting. If it is credential sprawl, budget enforcement, or compliance, adopt a gateway and skip the routing features entirely. If it is the bill on mixed traffic, model choice anxiety, or reliability across providers, the fix is a layer that reads the request before deciding, which is what model routing does and what our smart routing was built for. If it is both, run both, in that order.
The test travels well as the market keeps renaming things: whatever the product calls itself, ask what is allowed to change the model. If the answer is a file you maintain, it is a gateway. If the answer is the request itself, it is a router.