Most discussion of model selection is about the happy path. You pick the model that gives the best answer, or the cheapest token, or the fastest first byte, and you ship it. That choice is real work and we have written about it in how to choose the right AI model. But it answers only one question: which model should serve this request when everything is working. Production has a second question that is just as important and gets far less attention: what happens when the model you chose does not respond.
The failures that matter here are rarely about model quality; they are operational. A provider returns a 429, the HTTP status for too many requests, because you crossed a rate limit during a traffic spike. An endpoint returns a 5xx, the family of server-side error codes, because the provider is having an incident, a request hangs and trips your timeout, or a model you depend on gets deprecated and starts returning errors on a date that was never on your calendar. None of these are solved by a smarter prompt or a higher-scoring model. They are solved by having somewhere else to send the request, which is what a fallback chain provides when it is organized well.
Routing and fallback solve different problems
It is worth separating two ideas that often get merged. Model routing is an optimization on the happy path: given a request, choose the candidate that best balances quality, cost, and latency, and send the traffic there. Fallback is a resilience mechanism on the failure path: given that the chosen candidate did not return a usable response, decide what to try next so the caller still gets an answer.
The two are complementary, not interchangeable. You can route well and have no fallback, in which case your careful model selection still surfaces a provider outage as a 500 to your user. You can have fallback with no real routing, by pinning a primary and a backup by hand. The combination is what production wants: route to the right model, and fall through to an acceptable one when the right one is unavailable. Keeping the two concepts distinct also keeps the design honest, because the criteria differ. Routing asks "which is best?" Fallback asks "which is good enough, right now, given the first choice is gone?"
The failures a single model exposes you to
Relying on one model from one provider concentrates several independent risks into a single point of failure:
- Rate limits (429). Every hosted provider caps requests and tokens per minute. A burst of traffic, a noisy neighbor on a shared key, or a retry storm can put you over the limit, and the provider sheds your load precisely when you need it most.
- Outages and server errors (5xx). Providers have incidents. A regional degradation or a bad deploy on their side becomes your downtime if you have nowhere else to go.
- Timeouts and latency spikes. Even when a provider is up, tail latency can blow past your deadline. A request that eventually succeeds after twenty seconds is, for an interactive feature, a failure.
- Capacity and cold starts. Newer or larger models, and self-hosted endpoints that scale to zero, can reject or stall requests under load or on the first call after idle.
- Content-policy refusals. A provider's safety layer can decline a request that a different model would serve. This one is subtle, because it is a terminal failure on that model, not a transient one.
- Model deprecation and version churn. Models get sunset on a schedule: a snapshot you pinned for reproducibility stops being served, and an endpoint you have not touched in months starts failing. The cadence of new releases makes this a standing tax, not a one-time event, and its quiet counterpart, aliases re-pointed without any error at all, is worse precisely because no fallback ever triggers.
A fallback chain addresses every item on this list except the terminal refusal, and it does so with the same mechanism: when the current candidate cannot serve the request, move to the next one.
What a fallback chain is
A fallback chain is an ordered list of candidates. The request tries the first. If that attempt fails for a retryable reason, it advances to the second, and so on. The chain terminates the moment an attempt succeeds, and if every candidate is exhausted it returns a clear error, a 503 (service unavailable), rather than a silent empty response.
The candidates can sit at three levels, and a good chain often mixes them. The same model on a different provider protects against one provider's outage. A different model with comparable capability protects against a model-specific deprecation or refusal. A cheaper or smaller model as the final rung protects against a broad capacity crunch, accepting some quality loss to stay available. The ordering encodes your priorities: best first, acceptable last, and nothing in the chain that you would be unwilling to actually serve.
The hard parts
Writing a loop that tries the next item is easy. The reason fallback is worth treating as a design problem is that a naive chain can make reliability and economics worse, not better. The details below are where the real work is.
Retryable versus terminal failures. The single most important distinction. A 429, a 5xx, a timeout, or a capacity error is transient: the next candidate has a good chance of succeeding. A 400, a 401, or a content-policy refusal is terminal: it will fail the same way everywhere, so falling through wastes a budget you do not have and runs up cost for no benefit. The chain must classify the error before it decides to advance.
Per-attempt deadlines. If each candidate gets the full request timeout, a three-deep chain can take three times as long as a single call before it gives up, which is the opposite of resilience for an interactive feature. Each attempt needs its own slice of the overall budget so the chain as a whole stays inside one deadline, never the sum of every timeout. The diagram's footnote makes this point because it is the most common way a fallback chain degrades the experience it was meant to protect.
Idempotency and double billing. A request that times out may have already started, or even finished, server-side. If you advance and the original also completes, you can pay twice and, worse, deliver two different answers. The chain needs an idempotency key so a settle or charge happens once, keyed to the request rather than to the attempt.
Capability parity. A fallback is only safe if it can do what the primary did. If your code expects tool calls or a strict JSON shape, a backup that does not support those features will return something your pipeline cannot parse, turning an availability failure into a correctness failure. Parity covers the context window too: a backup with a smaller window will reject the same prompt the primary accepted.
The quality cliff. "Any model that responds" is not the bar. A fallback that returns a much weaker answer can be worse than a clean error, because the caller cannot tell the difference and ships the bad output downstream. Every rung in the chain should be one you are genuinely willing to serve, not just one that happens to be reachable.
Cost awareness. A sustained outage on a cheap primary will route real volume to the backup. If the backup is a frontier model, the chain quietly multiplies your per-request cost for the duration of the incident. Fallbacks should be acceptable on price as well as quality, and budgets should account for the failure path, not only the happy one; the reliability tax is one of the hidden costs that never appears on a price sheet.
Observability and auditability. When a chain advances, you need to know it happened, which candidate served the request, and why the previous one failed. Without that, a slow degradation looks like silence: the feature still works, the bill creeps up, and the primary has been failing for a week with no one the wiser. The response should carry the served model and the reason the chain moved, so the behavior is inspectable rather than a black box.
Fallback, retry, and circuit breakers are not the same
These three get conflated, and they compose better when kept distinct. A retry tries the same candidate again, with backoff and jitter, on the assumption the failure was momentary. Fallback tries a different candidate, on the assumption this one is not coming back soon enough. A circuit breaker stops sending traffic to a candidate that has been failing, for a cooldown period, so you are not paying the timeout tax on every request to an endpoint that is plainly down. This is also the standard defense against cascading failure, where retries into an overloaded provider deepen the very outage you are trying to route around.
The layered behavior is: one short retry against the primary for a transient blip, advance to the next candidate if it persists, and a circuit breaker in front of each candidate so a known-bad endpoint is skipped outright rather than retried into the ground. Skipping a tripped endpoint is also what keeps the per-attempt deadlines meaningful, because you do not burn budget waiting on a provider you already know is failing.
Where fallback belongs in the stack
You can implement all of this in the application, and for a single feature calling one or two models that is reasonable. It stops being reasonable as the surface grows. Retry logic, provider clients, error taxonomies, and timeout discipline end up copied across every service, each team's version slightly different, and the one place you most want a single answer, which model actually served this request, is exactly the place that fragments.
A routing or gateway layer consolidates the chain. The credentials, the per-attempt deadlines, the retryable-versus-terminal classification, the circuit breakers, and the observability live in one place, and every call inherits the same behavior. The cost is an extra hop in the path, which most teams past a couple of models find a worthwhile trade for not reimplementing resilience in every codebase.
How Inferbase handles fallback
Inferbase is a route-and-serve layer that puts this behavior behind a single endpoint. You call one OpenAI-compatible API, and behind it sits a pool of models spread across several providers. For each request, Inferbase serves from your pool, and if that attempt fails for a retryable reason, it advances through the remaining models under one request deadline. It is the fallback chain described above, run on your behalf rather than written into your application.
A handful of properties matter more than the mechanism, and they map onto the hard parts above:
- A fallback that can serve the request. A failover only advances to a model that can handle the request as issued, with the context window, tools, and output format it needs, so an availability problem never becomes a parsing problem.
- One deadline, not a sum of timeouts. The chain runs inside a single request budget, so advancing across providers does not stack every timeout onto the caller.
- Settle once. Billing is keyed to the request, so a chain that advances still charges a single time and never double-bills for one answer.
- Auditable by default. Every response reports which model actually served it and why the chain advanced, so a provider degrading quietly surfaces in your logs instead of as silent cost creep.
- Your pool, your priorities. You choose which models are in the pool and whether to optimize for quality, cost, or latency, and Inferbase handles the provider clients, the retry and timeout discipline, and the failover.
The result the caller sees stays consistent: a request is served by an available model from your pool, under the priorities you set, and every response shows which model served it and why the chain moved.
What this means in practice
A model is a dependency, and a single dependency with no alternative is a single point of failure regardless of how good it is. The work of choosing the best model is necessary but not sufficient, because the best model still has rate limits, still has incidents, and will eventually be deprecated. A fallback chain turns those operational realities from outages into advances.
The chain that actually helps is the one that classifies its errors, bounds each attempt to a slice of the deadline, settles each request once, keeps capability and quality parity across its rungs, watches the cost of the failure path, and reports what it did. Build that discipline into your own chain, or use a layer that already has it, and a bad minute at one provider stays contained there rather than propagating into your application.
Fallback is also only half of a larger decision. Choosing where a request goes when the first choice fails is the failure-path twin of choosing where it goes in the first place, and the two share their machinery: candidate pools, capability checks, cost constraints, observability. That happy-path half is model routing, and LLM routing platforms exist because the two halves are worth solving once, together, behind a single endpoint.