Methodology

This page documents the Inferbase products and tools, and the methodology behind the data and estimates that sit underneath them. It covers managed inference and smart routing, the model evaluation tools, and the GPU capacity planner.

The boundary is deliberate. For the products built on the routing engine, the page explains what each one does and when to use it, not the internal models or weights that decide a route. For the data pipeline and the GPU capacity planner, where the methods are standard and worth checking, it gives the full detail.

Managed inference

Managed inference is a single API in front of many models. You send a request to one endpoint, and Inferbase handles model serving, provider failover, and usage accounting, so you do not maintain a separate integration for each provider. Today it runs fully managed, where Inferbase serves the chosen model. Two more delivery modes are coming soon: bring-your-own-key, where requests run through your own provider accounts, and routing-only, where Inferbase returns the routing decision and you execute it.

AI model smart routing

Smart routing chooses which model should answer each request rather than committing every request to one model. The decision runs in stages. A request is first classified along two axes: the task it represents, such as coding, summarization, analysis, or conversation, and its complexity, which separates prompts a small efficient model can answer well from those that warrant a larger frontier model. Classification narrows the question; it never picks the model. The catalog is then filtered on hard constraints, the capabilities the request needs, context fit, and current availability, and every remaining candidate is ranked on task-conditioned quality evidence, price, and latency, weighted by the priority you select: balanced, quality, cost, or latency. The winner is the model the evidence supports for that task, not a fixed mapping from a category to a model, and a model we have not independently evaluated for a task is treated as unknown rather than assumed. The effect is that easy requests reach inexpensive models and hard requests reach capable ones, without you maintaining that logic.

The classification model and the scoring weights are part of the routing engine and are not detailed here. What the product commits to is the behavior: a request is scored on the objective you choose, not on our margin. A simple prompt is routed to a cheap open model even when a more expensive one would earn us more.

Auto routing

In auto routing you do not name a model at all. The request goes to the automatic endpoint and the router selects a model per request from the full catalog of servable models. This suits heterogeneous traffic, a mix of easy and hard prompts, where no single model is the right default for everything.

Custom model pool

A custom model pool constrains routing to a set of models you define. Teams use this to keep routing within an approved list, one vetted for compliance, licensing, or data residency, or simply a shortlist they trust. The router still selects the best-fit model for each request, but only from the pool you allow.

Routing transparency

Every routed response reports which model handled the request and why. You see the candidates that were considered, the priority that was applied, and the trade-off that decided the route, so a routing choice can be understood and audited rather than taken on faith. This matters in production: when a request goes to a cheaper or a stronger model than you expected, the reason is visible rather than hidden behind a black box.

Routing preview

Routing preview replays a sample of your own traffic through the router without changing what you currently run. It reports which models the router would have selected and the projected effect on cost and quality, so you can evaluate routing against your real workload before moving any production traffic onto it.

Cost savings

The savings figures published on this site come from a single measured run rather than a model of what routing might achieve. This section states how that run was constructed, what it found, and what it does not establish, so the number can be judged rather than taken on faith.

The run

On 2026-07-06 a batch of 100 prompts was replayed through routing preview against a Claude Opus 4.8 baseline, priced at $11.25 per million tokens. The router moved 90 of the 100 prompts to a cheaper model. The remaining 10 stayed on the baseline because no cheaper candidate cleared the quality bar for that prompt, which the report records rather than hides. The blended rate across the batch fell to $2.87 per million tokens, a 74% smaller bill.

The prompt set

The 100 prompts were written as a deliberate difficulty gradient, not a sample chosen to flatter the result. They open with single-label work such as ticket classification, spam detection, and language identification, pass through summarization, rewriting, and short factual questions, and close with genuinely demanding prompts: designing an event-sourcing architecture with schema evolution and snapshotting, analyzing a concurrency bug and arguing two fixes against each other, and modelling a three-year build-versus-buy total cost of ownership. Measured across the set, the first quarter averages 112 characters and the last quarter 297.

The gradient matters, because a savings claim is only meaningful if the batch contains prompts a small model should lose. It did, and the results show it: classification, extraction, and rewriting saved over 98% each, while the largest family, open-ended generation, saved 71%, and ten prompts saved nothing at all.

What the router chose

The shape of that table is the argument. Cheap models absorbed the easy majority, a mid-priced model was held back for the hardest generation work, and the most expensive prompts never left the frontier baseline at all.

How much of this is routing

Not all of it, and the distinction is worth stating plainly. The engine separates the headline into two components that sum to it exactly:

74 points total  =  62 points modernization  +  12 points routing lift

Modernization is what a single one-time switch from Claude Opus 4.8 to one cheaper model would have captured, with no routing at all. Routing lift is what per-prompt routing added on top of that switch. A large baseline-to-catalog price gap produces a large modernization component, which is why a customer already running a mid-tier model should expect a smaller headline and a proportionally larger share of it from routing.

This decomposition appears in every routing preview report, not only here. Reporting the combined figure as a routing result would be claiming credit for a decision any team could make without us.

How quality was evaluated

Automatically, and by projection rather than by judgment. Each candidate was required to clear a quality bar derived from benchmark evidence for the specific task, scaled to the difficulty of the prompt, before it was eligible to be routed to. Where the catalog held no quality evidence for a candidate on that task, the candidate was excluded rather than assumed adequate.

This run did not include the judged validation pass, in which both models answer the same prompt and a blind judge compares the outputs. The correct reading of the result is therefore "at projected equal-or-better quality on benchmark evidence", not "verified identical output quality". Judged validation is being built, and the figures here will be restated when it lands.

What this does not establish

The result is one batch, one baseline, one date. It measures published rates rather than billed invoices, and the summary records input tokens only, so output-heavy workloads need not reproduce the ratio. Model prices and the served catalog have both moved since the run. Any single organization's traffic mix, baseline model, and quality requirements will produce a different number, which is what routing preview exists to measure on your own prompts rather than ours.

Model data and provenance

The catalog underneath every tool is assembled from four kinds of source. Each field carries the source it came from and the time it was recorded, so any value on a model page can be traced back to its origin rather than taken on trust.

Every field reaches a model page through the same pipeline, and no data is hand-edited on production models. Ingestion collects raw data into a staging layer, tagging each field with its origin and timestamp. Normalization reconciles values that disagree across sources using priority rules, so each model resolves to a single consistent record rather than a pile of conflicting claims. Enrichment then derives use-case labels and capability tags from that record. Because enrichment runs over the normalized data rather than being written by hand, the provenance chain stays intact from source to page.

sources -> ingestion -> normalization -> enrichment -> model record

Model recommendation and comparison

The recommender scores available models against the requirements you state: use case, context-window needs, and whether you weight quality, cost, speed, or privacy. It diversifies results across providers rather than returning four variants of one family, and it includes open-weight options alongside proprietary ones so the choice is not narrowed by default.

Comparison places models side by side on a common scale. Benchmark scores are normalized so that a number from one leaderboard is comparable to another, grouped by domain (reasoning, math, code, knowledge), and capabilities are mapped onto a single matrix that is consistent across providers. Both tools score algorithmically on data; they do not capture qualitative behavior, which is noted in the limitations below.

GPU capacity planning

The capacity planner answers a chain of dependent questions, how much VRAM a model needs, how many concurrent requests a GPU can hold, the throughput and latency that follow, and how many GPUs the deployment requires, from a GPU's published specifications rather than from benchmarks. This makes the estimates hardware-portable but analytical, and the limitations section is explicit about what that costs in accuracy.

Total memory is the sum of model weights, the KV cache for the batched workload, activation memory, and framework overhead. Only the usable fraction of advertised VRAM counts, since the driver, CUDA context, and fragmentation all reserve memory.

Total VRAM  = Weights + KV cache (batched) + Activations + Overhead
Usable VRAM = GPU VRAM x 0.88

The KV cache is usually the binding constraint at long context, and its size depends on the attention design, not just the parameter count. Grouped-query attention stores Key and Value per retained head. Latent attention (MLA, used by the DeepSeek family and Kimi K2) stores one compressed latent per layer instead, roughly three times smaller. Sliding-window attention (Gemma, gpt-oss) caps the cache at the window regardless of context length. The planner sizes each model for the architecture it actually uses.

GQA per token = 2 x layers x kv_heads x head_dim x bytes
MLA per token = layers x (kv_lora_rank + rope_dim) x bytes
window:         effective tokens = min(context, window) on windowed layers

Decode throughput follows the roofline model: the lower of a memory-bandwidth bound and a compute bound. For mixture-of-experts models the weights read per step rise from the active set toward the full pool as the batch spreads tokens across experts, so MoE decode does not scale as cleanly as the active-parameter count suggests. When a model exceeds one GPU, the planner finds the smallest tensor- or pipeline-parallel degree that fits and applies an interconnect penalty when NVLink is absent. It then sizes replicas to satisfy both the concurrency the workload holds and the request rate it sustains, taking the larger of the two.

decode tok/s = min( batch x bandwidth / weights_read , FLOPS / (2 x active_params) )
replicas     = max( ceil( concurrency / max_batch ),
                    ceil( target_rps / (per_replica_capacity x 0.70) ) )

The full derivation, including the architecture estimation table, framework overhead factors, and worked examples validated against vLLM, is in the GPU capacity planning methodology.

Known limitations

• Benchmark variability. Benchmark scores depend on prompting strategy, evaluation version, and sampling. We record officially reported scores, but real-world performance on a specific task can differ from any single leaderboard number.
• Pricing lag. Provider pricing changes without notice. We refresh regularly, but the authoritative figure is always the provider page at the moment you deploy.
• GPU estimates are analytical. The roofline model reports the hardware ceiling, not a measured result. Framework quality, kernel efficiency, scheduling, and workload shape all reduce achieved performance. Benchmark the chosen configuration before committing to a latency target.
• Recommendation cannot judge style. The recommender scores models on capabilities and benchmarks. It cannot weigh output style, refusal behavior, or domain-specific nuance, which still require human evaluation.

The methods here apply across the platform: the model catalog, comparison, and the GPU capacity planner.