Methodology

This page documents the Inferbase products and tools, and the methodology behind the data and estimates that sit underneath them. It covers the AI gateway and the LLM routing inside it, the measured savings run, the model data pipeline, the model evaluation tools, and the GPU capacity planner.

The boundary is deliberate. For the products built on the routing engine, the page explains what each one does and when to use it, not the internal models or weights that decide a route. For the data pipeline and the GPU capacity planner, where the methods are standard and worth checking, it gives the full detail.

The AI gateway

The gateway is a single OpenAI-compatible API in front of many models. You send a request to one endpoint, and Inferbase authenticates the key, checks its rate limit and spend cap and the project's budget, serves the model you named or the one the router chose, handles failover, and writes a usage receipt and, for a routed request, the decision, both retrievable by request id. Two delivery modes are live. Managed, where Inferbase serves the chosen model from a managed pool of hosts and bills you the host's token price plus a flat 6%. And bring-your-own-key, where you connect your OpenAI, Anthropic, Gemini or other provider key to a project and requests run on your account: we route and serve the request but the tokens are billed by your provider, so the only Inferbase charge is the routing fee. A third mode, routing-only, where Inferbase returns the decision and you execute the call yourself, is not offered today.

LLM routing

LLM routing (Smart Routing, in the product) chooses which model should answer each request rather than committing every request to one model. The decision runs in stages. A request is first classified along two axes: the kind of work it represents and its complexity, which separates prompts a small efficient model can answer well from those that warrant a larger frontier model. Classification narrows the question; it never picks the model. The catalog is then filtered on hard constraints, the capabilities the request needs, context fit, and current availability, and every remaining candidate is ranked on task-conditioned quality evidence, price, and latency, weighted by the priority you select: balanced, quality, cost, or latency. The winner is the model the evidence supports for that task, not a fixed mapping from a category to a model, and a model we have not independently evaluated for a task is treated as unknown rather than assumed. The effect is that easy requests reach inexpensive models and hard requests reach capable ones, without you maintaining that logic.

The classification model and the scoring weights are part of the routing engine and are not detailed here. What the product commits to is the behavior: a request is scored on the objective you choose, not on our margin. A simple prompt is routed to a cheap open model even when a more expensive one would earn us more.

Auto routing

In auto routing you do not name a model at all. You send model="auto" and the router selects a model per request from the servable models within the scope the key allows (the managed pool, your own connected keys, or both) that clear the quality floor of the key's preset. This suits heterogeneous traffic, a mix of easy and hard prompts, where no single model is the right default for everything.

Custom model pool

A custom model pool constrains routing to a set of models you define. Teams use this to keep routing within an approved list, one vetted for compliance, licensing, or data residency, or simply a shortlist they trust. The router still selects the best-fit model for each request, but only from the pool you allow.

Routing transparency

Every routed response reports which model handled the request and why. You see the candidates that were considered, the priority that was applied, and the trade-off that decided the route, so a routing choice can be understood and audited rather than taken on faith. This matters in production: when a request goes to a cheaper or a stronger model than you expected, the reason is visible rather than hidden behind a black box.

Routing preview

Routing preview replays a sample of your own traffic through the router without changing what you currently run. It reports which models the router would have selected and the projected effect on cost and quality, so you can evaluate routing against your real workload before moving any production traffic onto it.

Cost savings

This site no longer publishes a savings percentage. It published one until 2026-09-18, drawn from a single measured run, and it was withdrawn when its baseline stopped being substantiable: the price that anchored it was a catalog row that has since been removed, and the only price the catalog now holds for that same model would move the headline by more than thirty points. A number whose baseline can move that far was never measuring routing. What the run does still support is described below, because the run happened and its per-model results stand on rates we pay.

The run

On 2026-07-06 a batch of 100 prompts was replayed through routing preview against a Claude Opus 4.8 baseline. The router moved 90 of the 100 prompts to a cheaper model. The remaining 10 stayed on the baseline because no cheaper candidate cleared the quality bar for that prompt, which the report records rather than hides. The blended rate across the batch was $2.87 per million tokens. The headline percentage that used to follow this sentence is gone, because it was a ratio against a baseline price we can no longer stand behind.

The prompt set

The 100 prompts were written as a deliberate difficulty gradient, not a sample chosen to flatter the result. They open with single-label work such as ticket classification, spam detection, and language identification, pass through summarization, rewriting, and short factual questions, and close with genuinely demanding prompts: designing an event-sourcing architecture with schema evolution and snapshotting, analyzing a concurrency bug and arguing two fixes against each other, and modelling a three-year build-versus-buy total cost of ownership. Measured across the set, the first quarter averages 112 characters and the last quarter 297.

The gradient matters, because a savings claim is only meaningful if the batch contains prompts a small model should lose. It did, and the results show it: the cheapest families moved almost entirely off the baseline, the largest family (open-ended generation) moved partly, and ten prompts did not move at all.

What the router chose

The shape of that table is the argument. Cheap models absorbed the easy majority, a mid-priced model was held back for the hardest generation work, and the most expensive prompts never left the frontier baseline at all.

How much of this is routing

Not all of it, and that was true even when the headline existed. The engine splits any savings figure into two components that sum to it exactly: modernization, what a single one-time switch to one cheaper model would have captured with no routing at all, and routing lift, what per-prompt routing adds on top of that switch. In the 2026-07-06 run, roughly five sixths of the headline was modernization.

That ratio is the reason the withdrawn figure was a poor advertisement for routing and a good argument for reading the decomposition. It also explains why a team already running a mid-tier model should expect a smaller total and a proportionally larger share of it from routing.

The decomposition appears in every routing preview report, which is where a number for YOUR workload comes from: run your own traffic through it and read the split. That is a measurement of your bill, which a percentage from our batch never was.

How quality was evaluated

Automatically, and by projection rather than by judgment. Each candidate was required to clear a quality bar derived from benchmark evidence for the specific task, scaled to the difficulty of the prompt, before it was eligible to be routed to. Where the catalog held no quality evidence for a candidate on that task, the candidate was excluded rather than assumed adequate.

This run did not include the judged validation pass, in which both models answer the same prompt and a blind judge compares the outputs. The correct reading of the result is therefore "at projected equal-or-better quality on benchmark evidence", not "verified identical output quality". Judged validation is being built, and the figures here will be restated when it lands.

What this does not establish

The result is one batch, one baseline, one date. It measures published rates rather than billed invoices, and the summary records input tokens only, so output-heavy workloads need not reproduce the ratio. Model prices and the served catalog have both moved since the run. Any single organization's traffic mix, baseline model, and quality requirements will produce a different number, which is what routing preview exists to measure on your own prompts rather than ours.

Model data and provenance

The catalog underneath every tool is assembled from four kinds of source. Each field carries the source it came from and the time it was recorded, so any value on a model page can be traced back to its origin rather than taken on trust.

Every field reaches a model page through the same pipeline. An administrator can override a field, but the override is stored as its own source, marked as such, and wins over the API sources without erasing them. Ingestion collects raw data into a staging layer, tagging each field with its origin and timestamp. Normalization reconciles values that disagree across sources using priority rules, so each model resolves to a single consistent record rather than a pile of conflicting claims. Enrichment then derives use-case labels and capability tags from that record. Because enrichment runs over the normalized data rather than being written by hand, the provenance chain stays intact from source to page.

sources -> ingestion -> normalization -> enrichment -> model record

Model recommendation and comparison

The recommender scores available models against the requirements you state: use case, context-window needs, and whether you weight quality, cost, speed, or privacy. It diversifies results across providers rather than returning four variants of one family, and it includes open-weight options alongside proprietary ones so the choice is not narrowed by default.

Comparison places models side by side on a common scale. Benchmark scores are normalized so that a number from one leaderboard is comparable to another, grouped by domain (reasoning, math, code, knowledge), and capabilities are mapped onto a single matrix that is consistent across providers. Both tools score algorithmically on data; they do not capture qualitative behavior, which is noted in the limitations below.

GPU capacity planning

The capacity planner answers a chain of dependent questions, how much VRAM a model needs, how many concurrent requests a GPU can hold, the throughput and latency that follow, and how many GPUs the deployment requires, from a GPU's published specifications rather than from benchmarks. This makes the estimates hardware-portable but analytical, and the limitations section is explicit about what that costs in accuracy.

Total memory is the sum of model weights, the KV cache for the batched workload, activation memory, and framework overhead. Only the usable fraction of advertised VRAM counts, since the driver, CUDA context, and fragmentation all reserve memory.

Total VRAM  = Weights + KV cache (batched) + Activations + Overhead
Usable VRAM = GPU VRAM x 0.88

The KV cache is usually the binding constraint at long context, and its size depends on the attention design, not just the parameter count. Grouped-query attention stores Key and Value per retained head. Latent attention (MLA, used by the DeepSeek family and Kimi K2) stores one compressed latent per layer instead, roughly three times smaller. Sliding-window attention (Gemma, gpt-oss) caps the cache at the window regardless of context length. The planner sizes each model for the architecture it actually uses.

GQA per token = 2 x layers x kv_heads x head_dim x bytes
MLA per token = layers x (kv_lora_rank + rope_dim) x bytes
window:         effective tokens = min(context, window) on windowed layers

Decode throughput follows the roofline model: the lower of a memory-bandwidth bound and a compute bound. For mixture-of-experts models the weights read per step rise from the active set toward the full pool as the batch spreads tokens across experts, so MoE decode does not scale as cleanly as the active-parameter count suggests. When a model exceeds one GPU, the planner finds the smallest tensor- or pipeline-parallel degree that fits and applies an interconnect penalty when NVLink is absent. It then sizes replicas to satisfy both the concurrency the workload holds and the request rate it sustains, taking the larger of the two.

decode tok/s = min( batch x bandwidth / weights_read , FLOPS / (2 x active_params) )
replicas     = max( ceil( concurrency / max_batch ),
                    ceil( target_rps / (per_replica_capacity x 0.70) ) )

The full derivation, including the architecture estimation table, framework overhead factors, and worked examples validated against vLLM, is in the GPU capacity planning methodology.

Known limitations

• Benchmark variability. Benchmark scores depend on prompting strategy, evaluation version, and sampling. We record officially reported scores, but real-world performance on a specific task can differ from any single leaderboard number.
• Pricing lag. Provider pricing changes without notice. We refresh regularly, but the authoritative figure is always the provider page at the moment you deploy.
• GPU estimates are analytical. The roofline model reports the hardware ceiling, not a measured result. Framework quality, kernel efficiency, scheduling, and workload shape all reduce achieved performance. Benchmark the chosen configuration before committing to a latency target.
• Recommendation cannot judge style. The recommender scores models on capabilities and benchmarks. It cannot weigh output style, refusal behavior, or domain-specific nuance, which still require human evaluation.

The methods here apply across the platform: the model catalog, comparison, and the GPU capacity planner.