How Our GPU Capacity Planning Calculator Works

Deploying a self-hosted AI model requires answering a series of interdependent infrastructure questions: how much VRAM the model needs, how many concurrent requests a given GPU can sustain, what latency the end user will experience, and whether a single GPU suffices or the workload must be distributed across several. These questions are not independent of one another, as the memory budget constrains batch capacity, which in turn determines throughput and latency. The GPU Capacity Planning tool is designed to solve them together.

This document provides a detailed explanation of every formula, assumption, and estimation method used by the calculator. Where our estimates diverge from real-world results, the reasons are stated explicitly. For a practical walkthrough of using the tool to size GPUs for specific models, see our GPU sizing guide for LLM inference.

1. Model weight memory

The memory required to store a model's parameters is deterministic. Each parameter occupies a fixed number of bytes determined by the selected precision, and the total is a straightforward multiplication:

Model Weights (GB) = Parameters x Bytes per Parameter / 1,073,741,824

FP4 is the Blackwell-native 4-bit float, and MXFP4 (micro-scaled FP4 with per-block scales) is the format gpt-oss ships in. Both occupy half a byte per parameter, the same footprint as INT4, but recover quality through hardware-accelerated scaling on B200-class GPUs.

For fine-tuning use cases, the optimizer maintains additional state that must also reside in VRAM:

• LoRA/QLoRA: +0.5 bytes per parameter (adapter weights and optimizer state for approximately 5-10% of parameters)
• Full fine-tuning: +8 bytes per parameter (Adam optimizer requires 4 bytes for momentum and 4 bytes for variance, both stored in FP32)

2. Architecture estimation

The model's parameter count alone is not sufficient to calculate KV cache memory. That calculation also requires knowledge of the model's internal architecture: the number of layers, attention heads, head dimensions, and whether the model uses Grouped Query Attention. When a user selects a model from the Inferbase catalog, we know its parameter count but may not have its full architecture specification. In those cases, we estimate the architecture by interpolating between known reference points:

These anchor points are derived from published Llama, Qwen, and Mistral architectures. For models whose parameter count falls between two anchors, the tool interpolates linearly. Models above 3B parameters are assumed to use Grouped Query Attention (GQA) with 8 KV heads, which substantially reduces KV cache size compared to Multi-Head Attention.

Known models override the estimate. For well-documented models the tool carries the real architecture facts instead of interpolating: the Mixture-of-Experts total-versus-active split (memory uses the full expert count, compute uses the active count), the exact layer count, and the attention variant. This matters most for the DeepSeek family and Kimi K2, which use Multi-head Latent Attention (covered in the next section). Models outside that curated set fall back to the interpolation above, which is explicitly an approximation.

3. KV cache memory

During autoregressive inference, the model stores Key and Value tensors for every token in the context window. This accumulated state is the KV cache, and its size is the primary factor determining how many concurrent requests a GPU can serve.

The per-token KV cache size is calculated as:

KV Cache per Token (bytes) = 2 x Layers x KV Heads x Head Dim x Bytes per Element

The factor of 2 accounts for both Key and Value tensors. This formula applies to Multi-Head and Grouped Query Attention.

Multi-head Latent Attention (MLA), used by the DeepSeek V2/V3 family and Kimi K2, caches differently. Instead of storing Key and Value per head, it stores a single compressed latent plus a small decoupled positional key per layer, shared across all heads:

MLA KV Cache per Token (bytes) = Layers x (KV LoRA Rank + RoPE Head Dim) x Bytes per Element

For DeepSeek V3 (61 layers, 512 latent rank, 64 RoPE dim) this is roughly three times smaller than the GQA formula would predict at the same context length, which is why MLA models pack far more concurrent requests into the same VRAM. The tool applies this formula automatically for models it knows use MLA.

A second architecture-specific case is sliding-window attention, used by Gemma and gpt-oss. A windowed layer only attends to the most recent W tokens, so its KV cache stops growing once the context exceeds the window, no matter how long the prompt gets. Many of these models interleave windowed and full-attention layers (Gemma 3 runs five local layers per global layer; gpt-oss alternates one-to-one), so the tool blends the two by their ratio:

Effective KV Tokens = GlobalRatio x Context + (1 - GlobalRatio) x min(Context, Window)

The practical effect appears only at long context. A pure sliding-window model at 128K context with a 1,024-token window caches as if the context were 1,024 tokens, a difference of two orders of magnitude against a full-attention model.

An important distinction: KV cache precision is independent of model weight quantization. By default, the KV cache is stored in FP16 (2 bytes per element) even when model weights are quantized to INT4. Users can optionally select FP8 or INT8 KV cache precision to halve the cache size, provided their serving framework supports it.

The total KV cache memory for a single request is then:

KV Cache per Request (GB) = KV Bytes per Token x Context Length / 1,073,741,824

To illustrate the scale of this cost, consider Llama 70B with a 4,096 token context length:

• KV bytes per token = 2 x 80 layers x 8 KV heads x 128 head dim x 2 bytes = 327,680 bytes
• Per request = 327,680 x 4,096 / 1,073,741,824 = 1.25 GB

Each concurrent user therefore adds approximately 1.25 GB of VRAM usage. At 128K context, this jumps to approximately 40 GB per request, which is why long-context workloads are particularly constrained by KV cache memory.

KV cache strategy

The serving framework's KV cache management strategy affects actual memory consumption. The tool applies a memory efficiency multiplier based on the selected strategy:

4. Activation memory

Beyond model weights and the KV cache, the GPU must also allocate VRAM for activation memory: the intermediate tensors produced during the forward pass (attention scores, feed-forward network intermediates). For inference workloads, these allocations are transient, created during each decode step and freed immediately after.

The tool estimates activation memory as a fraction of model weight memory:

Activation Memory (GB) = Model Weights (GB) x Activation Fraction

The fraction varies by use case because different workloads hold different amounts of intermediate state:

The use of a percentage rather than an exact calculation is a deliberate simplification. Exact activation memory depends on batch size, sequence length, the specific attention implementation, and framework internals. Modern serving frameworks such as vLLM and TGI manage activations dynamically through CUDA's caching allocator, making the precise value difficult to predict from architecture parameters alone. The percentages represent conservative upper bounds for steady-state overhead.

5. Framework overhead

Each serving framework carries a baseline memory footprint for CUDA contexts, runtime state, tokenizer caches, and internal data structures. The tool models this as a multiplicative factor applied to the sum of all other memory components:

Overhead (GB) = (Weights + KV Cache + Activations) x (Overhead Factor - 1)

The tool selects the larger of the quantization overhead factor or the framework overhead factor:

A multiplicative model was chosen over a fixed-overhead model because real-world overhead has both fixed and variable components. The CUDA context itself is approximately 0.5 GB (fixed), while memory fragmentation accounts for 2-5% (variable). At the model sizes where GPU capacity planning is most relevant (7B parameters and above), the multiplicative approach approximates both components adequately. For very small models under 1B parameters, this approach slightly underestimates overhead.

6. Total VRAM and usable fraction

The total VRAM requirement is the sum of all components:

Total VRAM Needed = Weights + KV Cache (batched) + Activations + Overhead

However, not all of a GPU's advertised VRAM is available for model serving. The operating system, CUDA driver, and display manager each reserve memory. The tool accounts for this with a fixed utilization ceiling:

Usable VRAM = GPU VRAM x 0.88 (88%)

This is intentionally more conservative than the default gpu_memory_utilization of 0.95 used by vLLM. The 12% margin accounts for three sources of unavailable memory:

• CUDA context and driver overhead (approximately 2-3%)
• Memory fragmentation (approximately 3-5%)
• Headroom for allocation spikes during request bursts (approximately 2-4%)

7. Tensor parallelism (TP) determination

When a model's memory requirements exceed what a single GPU can provide, the tool distributes the model across multiple GPUs using tensor parallelism. The goal is to find the minimum TP degree (from the set 1, 2, 4, 8, 16) at which the per-GPU memory budget is sufficient:

Per-GPU Fixed Memory = (Model Weights + Activations) / TP Degree
Per-GPU KV Headroom  = Usable VRAM / Overhead - Per-GPU Fixed Memory
Max Batch Size       = KV Headroom / (KV per Request / TP Degree)

The overhead factor is applied to the VRAM budget rather than the memory consumption. This prevents a scenario where the model weights and activations fit within the GPU, but the combined KV cache and framework overhead push total usage beyond the available VRAM.

TP with NVLink vs PCIe: Tensor parallelism requires inter-GPU communication at every transformer layer (an all-reduce operation). When the selected GPU lacks a high-bandwidth interconnect such as NVLink or Infinity Fabric, the tool applies a 60% bandwidth penalty to throughput estimates. This reflects the real-world performance degradation of tensor parallelism over PCIe, where the all-reduce synchronization at each layer becomes a throughput bottleneck.

8. Batch capacity

Once the fixed memory components (weights, activations) are allocated, the remaining VRAM determines how many concurrent requests the GPU can serve. Each additional request in the batch requires its own KV cache allocation:

KV Headroom (GB)   = Usable VRAM per GPU - Fixed Memory per GPU
Max Batch Size     = floor(KV Headroom / KV Cache per Request per GPU)
Effective Batch    = min(Max Batch, Concurrency)

9. Throughput: roofline model

The tool estimates throughput using the roofline model, a physics-based analytical framework that bounds performance by the hardware's memory bandwidth and compute capacity. This approach derives estimates from the GPU's specifications rather than extrapolating from benchmarks, which makes the results hardware-portable but necessarily theoretical.

Decode throughput (autoregressive generation)

Each decode step reads all model weights from GPU memory and produces one token per batch member. Depending on the batch size, throughput is limited by one of two hardware constraints:

Memory-bound throughput = Batch Size x Bandwidth / Model Size per GPU
Compute-bound ceiling   = FLOPS / (2 x Parameters per GPU)
Actual throughput       = min(Memory-bound, Compute-bound)

• Memory-bound (most common): throughput scales linearly with batch size because the GPU spends most of its time reading weights from HBM. Batching amortizes this memory access cost across multiple requests.
• Compute-bound (very large batches): throughput reaches a ceiling determined by the GPU's floating-point operations per second.

For Mixture-of-Experts models, "Model Size per GPU" is not fixed. A single request reads one active-expert set, but as the batch grows, different tokens route to different experts, so the union of weights touched per decode step climbs toward the full pool. Assuming roughly uniform routing, the touched fraction is approximately 1 - (1 - active/total)^batch. The tool models this, so an MoE model's decode throughput does not scale with batch as cleanly as its active-parameter count alone would suggest. The compute ceiling still uses the active count, since the floating-point work per token is genuinely active-only.

The crossover batch size, where the workload transitions from memory-bound to compute-bound, is FLOPS / Bandwidth. For the A100, this crossover occurs at approximately 153 concurrent requests; for the H100, at approximately 591. Most production inference workloads operate well within the memory-bound regime.

The tool applies a practical utilization factor to the GPU's theoretical peak bandwidth to account for real-world memory access patterns:

• Pessimistic: 70% (fragmented access patterns, worst case)
• Expected: 80% (typical for LLM inference)
• Optimistic: 90% (optimal memory access, best case)

These three utilization levels produce the min/expected/max range reported in the tool's output.

Example: Llama 8B FP16 on A100 80GB (single GPU, batch size 1):

• Effective bandwidth = 2,039 GB/s x 80% = 1,631 GB/s
• Model size = 14.9 GB
• Throughput = 1 x 1,631 / 14.9 = 109 tokens/sec
• Real-world vLLM benchmarks for this configuration report 80-120 tokens/sec

Prefill throughput (prompt processing)

Unlike decode, prefill is compute-bound because all input tokens are processed in parallel through a single forward pass:

Prefill throughput = FLOPS x TP / (2 x Total Parameters)

This represents how quickly the GPU processes the input prompt before autoregressive generation begins. With tensor parallelism, each GPU handles 1/TP of the computation, so total prefill throughput scales linearly with the TP degree. Pipeline parallelism deliberately does not appear in this formula: a single request traverses the pipeline stages one after another, so splitting the layers across stages does not shorten that one request's prefill (and therefore does not shorten its time to first token). Pipeline parallelism raises aggregate throughput, not single-request latency.

Pipeline parallelism and the pipeline bubble

When a model is split into pipeline stages rather than sharded within each layer, the stages form an assembly line: stage two cannot start until stage one hands off its activations. With only one request in flight, every stage but one sits idle at any instant, so a two-stage pipeline runs no faster than a single GPU. Throughput recovers only when several requests (microbatches) are in flight at once, so that while stage two works on the first request, stage one is already working on the second.

The tool models this with the standard pipeline-bubble efficiency. For a pipeline of D stages fed m in-flight microbatches (the per-replica batch), the stages are collectively idle for D - 1 of every m + D - 1 steps, giving:

Pipeline efficiency = m / (m + D - 1)

This is applied to the decode throughput a pipeline-parallel replica would otherwise reach by aggregating bandwidth across its stages. The behavior is both batch-aware and depth-aware: at batch one the efficiency is 1 / D (no speedup over a single GPU), at a large batch it approaches one (near-perfect aggregation), and deeper pipelines pay a larger bubble at any given batch. A single flat efficiency factor cannot capture this, since it would over-credit pipeline parallelism at low concurrency and under-credit it at high concurrency.

10. Latency estimation

Latency estimates are derived directly from the throughput calculations described above, not from empirical multipliers or heuristics.

Time to first token (TTFT)

TTFT = Input Tokens / Prefill Throughput

TTFT measures how long the user waits before the first generated token appears. It is determined entirely by prefill throughput and input length: longer prompts produce proportionally longer TTFT.

Time between tokens (TBT)

TBT = Per-replica Batch / Decode Throughput

Under continuous batching, one replica's decode bandwidth is shared across the requests it serves simultaneously (its effective batch, not the full workload concurrency, which is spread across replicas). Each user receives one token per decode step, but the decode step serves the whole batch at once. Since the decode throughput in the numerator is itself measured at that same batch, this resolves to the per-stream token interval: each user's token speed is governed by how fast one replica streams, not by how many replicas the fleet runs.

Total latency

Total = TTFT + Output Tokens x TBT

Percentiles

The tool reports three latency levels, each based on a different bandwidth utilization assumption:

• P50: expected throughput (80% bandwidth utilization)
• P95: pessimistic throughput (70% bandwidth utilization)
• P99: P95 x 1.3 (additional variance for tail latency from scheduling delays, memory allocation, and garbage collection pauses)

These are not true statistical percentiles derived from a distribution of observed latencies. They represent the performance range that arises from bandwidth utilization variance alone. True percentiles would require modeling request arrival patterns, sequence length distributions, and framework-specific scheduling behavior, none of which the tool attempts.

11. Scoring and ranking

The tool ranks GPU configurations using a multi-criteria scoring system.

The default ranking ("best fit") applies the following priority order:

1. Meets latency target: configurations that satisfy the user's latency constraint rank above those that do not
2. Fewer GPUs: single-GPU configurations rank above multi-GPU setups, reflecting the operational simplicity of fewer devices
3. Higher throughput: at the same GPU count, the faster configuration ranks higher, since raw performance matters more than packing memory tightly
4. Higher VRAM utilization: the final tiebreaker, preferring the GPU that is best matched in size when throughput is equal

The tool ranks on the engineering dimensions above. It does not rank on or display cost: it answers which hardware fits and how fast it runs, and leaves the spend comparison (self-host versus a hosted API) to you.

12. Replica sizing for a target load

A deployment is sized as a number of identical replicas behind a load balancer, where one replica is the per-GPU configuration (including its tensor- or pipeline-parallel degree). The workload imposes two independent constraints, and the tool takes the larger:

Concurrency replicas = ceil(Concurrency / Max Batch per replica)
Throughput replicas  = ceil(Target Requests per Second / (Per-replica Capacity x Target Utilization))
Replicas             = max(Concurrency replicas, Throughput replicas)

The first constraint holds the peak number of simultaneous requests: a replica can keep only so many sequences in its KV-cache headroom (its max batch), so enough replicas are needed to give every concurrent request a slot. The second holds the request rate: a replica's capacity is its sustained requests per second measured at that full batch (decode throughput at max batch divided by the average output length), not at the smaller operating batch a light load happens to run. Throughput replicas scale out only once the target exceeds the safe operating point, so the count rises smoothly rather than jumping the instant the target crosses raw capacity.

Target utilization defaults to 70%. Planning a replica to run at 100% of its sustained capacity leaves no headroom for traffic spikes or for a replica dropping out, so the tool sizes the rate constraint against a steady-state ceiling below saturation. The result is reported as a replica count and the total GPU count across all replicas (replicas times the per-replica GPU count), which is the number to provision for the stated load. The high-availability multiplier then applies on top, adding redundant capacity for failover rather than for throughput.

When a replica can hold only one or two concurrent requests, for example a large model at full precision that barely fits and leaves almost no KV-cache headroom, the concurrency constraint forces a large replica count. The tool flags this as KV-cache-starved and points to the levers that raise batch capacity (lower weight precision, an FP8 KV cache, a larger-VRAM GPU, or a shorter context) rather than presenting the inflated fleet as the intended answer.

Known limitations

1. Architecture estimation: For models outside the curated registry, the tool interpolates architectural parameters from a set of known reference points. Models with non-standard designs (unusually deep or shallow networks, or attention variants the tool does not recognize) may have KV cache requirements that differ from the interpolated values. Known Mixture-of-Experts and MLA models are handled explicitly and are not affected by this.

2. Replica sizing assumes uniform load balancing: The tool recommends a replica count for a target request rate, but it assumes traffic spreads evenly across replicas and that each replica reaches its modeled per-instance capacity. Real load balancers, uneven request sizes, and cold replicas all reduce the achieved per-replica throughput, so treat the replica count as a floor, not a ceiling.

3. Throughput is theoretical: The roofline model provides the hardware's theoretical capability. Real-world throughput depends on framework implementation quality, CUDA kernel efficiency, the specific attention algorithm used, and numerous other runtime factors. In practice, expect 70-90% of the tool's estimate.

4. Latency percentiles are approximate: True P95/P99 latency is a function of queuing theory, request arrival patterns, and framework scheduling, none of which the tool models. The reported percentiles reflect bandwidth utilization variance only.

5. No memory fragmentation modeling: CUDA memory fragmentation can reduce effective VRAM by 5-15% over the lifetime of a long-running service. The 88% usable VRAM fraction partially accounts for this, but services that have been running for extended periods may experience higher fragmentation than the model assumes.

6. Interconnect cost is modeled coarsely: Tensor parallelism without a fast interconnect (no NVLink or Infinity Fabric) is penalized 40%, but tensor parallelism over NVLink still carries some all-reduce cost that the tool does not subtract, so large tensor-parallel configurations read slightly optimistic. Pipeline parallelism, which transfers only one activation per stage boundary, carries no such interconnect penalty; its distinct cost, the pipeline bubble, is covered in the throughput section above.

7. Compute ceiling uses FP16 throughput: The roofline compute bound is taken from each GPU's published FP16 TFLOPS regardless of the quantization you select. Low-precision formats (FP8, FP4) run faster on Blackwell-class hardware, so the compute ceiling can be conservative. This rarely changes the result, because decode is almost always memory-bound rather than compute-bound.

---

Try the GPU Capacity Planning Calculator with your own model parameters. For a practical guide on choosing GPUs for production inference, see GPU Sizing Guide for LLM Inference.