Memory, throughput, latency and GPU count for self-hosting any model, from its architecture and the card's published figures.
A physics-based model over published GPU figures: a memory budget for the weights and cache, then bandwidth and compute bounds for speed.
Memory comes first. The weights take the parameter count times the bytes per parameter at the precision you serve; the KV cache takes the architecture's per-token cache times the context length times the requests in flight, with multi-head latent attention, mixture-of-experts and sliding-window layers each sized by their own rule; activations and runtime overhead sit on top.
Speed follows from the card. Generating a token reads the weights and the cache from memory, so decode is bounded by memory bandwidth; processing a prompt is bounded by compute. The planner applies the serving framework's efficiency and a utilization range to each bound and reports tokens per second and time to first token as a range rather than a point.
Replicas close the plan: your target request rate divided by one instance's sustained capacity at about 70 percent utilization, so a spike or a lost replica does not saturate the fleet. The figures are analytical, not benchmarked; expect 70 to 90 percent of them in practice.
Computed by the planner's own engine for a grouped-query architecture of 80 layers, estimated from the parameter count. Throughput and replicas follow from bandwidth and compute on the chosen card.
Companion tools that draw on the same catalog and routing engine.
How to calculate the memory an LLM needs, how the cache grows, and how many GPUs a request rate takes.
Add four parts. Weights: the parameter count times the bytes per parameter at the precision you serve, two bytes at FP16, one at INT8, half a byte at INT4. KV cache: the per-token cache size for the architecture, times the context length, times the requests served at once. Activations: a few percent of the weights during inference. Runtime overhead: the framework's own allocations, applied as a factor. The planner computes each line and shows them separately, so you can see which one is driving the GPU count.
For a grouped-query model the cache holds a key and a value per layer per KV head per token: two times the layer count times the KV heads times the head dimension, times the bytes per value. It grows linearly with context length and with the number of requests in flight, which is why a long-context model at high concurrency can need more memory for cache than for weights. Multi-head latent attention caches one compressed latent per layer instead and comes out several times smaller.
INT8 halves the weight memory of an FP16 model and INT4 quarters it, and the planner sizes at each precision you choose, KV cache precision separately. The quality cost of quantization is not modeled here: the planner is a capacity tool, and the catalog's benchmark evidence is measured at the precision the publisher ran. Size at the precision you intend to serve, then check that precision's quality on your own evaluation set.
The capacity planner uses a physics-based roofline model over published GPU specifications: memory bandwidth, compute throughput, VRAM, and the KV cache the model actually allocates, including latent-attention (MLA), mixture-of-experts, and sliding-window architectures. The numbers are analytical rather than benchmarked, so treat them as a credible starting point and expect 70 to 90 percent of the estimate in practice.
DeepSeek V2/V3 and Kimi K2 use Multi-head Latent Attention, which caches one compressed latent per layer instead of a key and value per head. The planner applies the MLA formula for these models, so their KV cache comes out roughly three times smaller than a grouped-query model at the same context length, and the recommended GPU count reflects that.
A Mixture-of-Experts model is sized on two figures: total parameters drive the memory budget, since every expert must stay resident in VRAM, while active parameters drive compute. The planner keeps these separate, so it neither under-provisions VRAM by treating the model as its smaller active size nor under-estimates throughput by treating it as its full size.
Yes. KV cache precision is independent of weight quantization: the cache defaults to FP16 but can be set to FP8 or INT8 to halve it on supported frameworks. PagedAttention and prefix caching are modeled as memory-efficiency multipliers, since they reduce the over-allocation a naive contiguous cache would require.
Divide your target requests per second by a single instance's sustained capacity, then add headroom. The planner reports the replica count and the total GPUs across replicas for the request rate you enter, sizing each replica to about 70 percent utilization so that a traffic spike or a lost replica does not saturate the fleet.
One gateway in front of every model, with your policies applied and every decision on record. Start with $5 of credit and 5,000 routing decisions a month, no card required.