By every published measure, intelligence has never been cheaper. Epoch AI's analysis of inference price trends found that the price of matching a fixed capability level has fallen between 9x and 900x per year depending on the task, with a median around 50x, and the trend accelerating rather than tapering after January 2024. Andreessen Horowitz called the pattern LLMflation: for equivalent capability, the cost of inference has been dropping roughly 10x every year since the GPT-3 era. Gartner projects the curve continuing through the decade.
And yet the conversation inside engineering teams has moved in the opposite direction. Inference budgets are rising, finance is asking questions, and the teams being asked cannot point to a price increase to explain it, because there mostly has not been one. The resolution to the paradox is arithmetic rather than mystery: a bill is price per token multiplied by tokens consumed, and while the first factor has been falling fast, the second has been rising faster. Understanding where the tokens go is now the core skill of managing LLM inference costs, and it is a different skill from shopping for cheaper rates.
The price side: real, fast, and unequal
The decline in per-token prices is not marketing. It is driven by hardware generations, inference-stack efficiency, model distillation, and intense competition among providers serving comparable open-weight models, a market we track daily across our model catalog. A capability that required a frontier model at frontier prices two years ago is routinely available today from mid-tier or small open models at a small fraction of the rate.
The decline is also unequal, which matters for budgeting. Epoch's data shows commodity-level capabilities falling fastest, while matching the frontier on the hardest tasks holds its price far better. The models at the top of the capability range continue to launch at premium rates, and reasoning-optimized variants price their output tokens well above their standard siblings. Prices fall steeply behind the frontier and slowly at it, so a workload pinned to the newest frontier model captures little of the deflation that headlines celebrate. The mechanics of what a listed price does and does not include are a separate problem, one we covered in The Hidden Costs of LLM APIs.
The consumption side: three multipliers
If prices per token fell 10x while your bill doubled, consumption grew roughly 20x. That growth did not come from traffic alone. Three structural shifts in how applications use models each multiplied tokens per task, and they compound where they overlap.
Reasoning tokens: paying for thinking
Reasoning models changed the shape of an LLM response. Instead of producing an answer directly, they generate chains of intermediate reasoning (largely invisible to the user) before the visible output, and those thinking tokens bill as output tokens, the most expensive kind. The volume scales with difficulty: an easy prompt may think briefly, while a hard one can produce thousands of reasoning tokens in front of a three-sentence answer. Token consumption becomes sensitive to the input in a way that fixed per-request budgets never anticipated.
The trade is often worth it, since test-time compute buys real quality on genuinely hard problems. The cost failure is applying it indiscriminately: routing a stream of routine requests to a reasoning model spends thinking tokens on prompts that a standard model would answer identically. We explain the mechanism in detail in What Are Reasoning Models.
Agent loops: the context tax, compounding
Agentic workloads multiply tokens through a different mechanism: repetition. A tool-calling agent works in steps, and each step re-sends the accumulated context, including the system prompt, conversation history, tool definitions, and every previous tool result. The context grows as the task progresses, so late steps are the most expensive, and the total input tokens for a task grow roughly with the square of the step count until context management intervenes.
Industry analyses of this pattern put agentic tasks at 5 to 30 times the token consumption of a comparable chatbot exchange, with complex multi-step work going far higher, and the same task varying widely in consumption between runs. The driver is usually input tokens rather than output, which surprises teams whose cost intuitions were formed on chat workloads. It is also why prefix caching discipline and context trimming have become first-order cost levers rather than optimizations; our guide to prompt caching covers why caches hit far less often than teams assume.
Context growth: because you can
The third multiplier is quieter. Context windows expanded from thousands of tokens to hundreds of thousands, with million-token windows now common, and applications absorbed the headroom. Retrieval pipelines that once carefully selected three relevant passages now ship thirty, whole documents ride along because trimming them takes engineering effort, and long-running conversations accumulate history that nobody prunes. Each decision is individually reasonable, and collectively they multiply the input tokens of every single request that follows. The window's economics, including why long contexts also degrade quality in the middle, are covered in What Is a Context Window.
Jevons, restated for tokens
Economists have a name for demand growing faster than efficiency improves: the Jevons paradox, coined for Victorian coal and perfectly at home in modern inference. Cheaper tokens do not merely make existing workloads cheaper; they make previously irrational workloads rational. At 2023 prices, nobody would run a 30-step autonomous agent that consumes two million tokens to file a support ticket summary. At current prices it is a product decision, and products are being built exactly that way.
This is why falling prices and rising budgets are not in tension: the price decline is the cause of the consumption growth, not a failed offset to it. Every 10x drop in price per token expands the frontier of what teams are willing to spend tokens on, and the expansion has consistently outrun the drop. Budgets are set against value delivered, and cheaper tokens let more value be attempted per task. The planning error is only in expecting the per-token curve to show up in the monthly total.
Price per token is the wrong unit
The practical consequence is that the number on the pricing page has become a weak predictor of what a workload costs. Two applications calling the same model at the same listed rate can differ by two orders of magnitude in cost per task, depending on reasoning budgets, agent step counts, context discipline, and cache hit rates. Comparing providers on per-token price while ignoring tokens per task optimizes the smaller factor.
The unit that predicts budgets is cost per task, or for product teams, cost per user action: the blended total of every model call, reasoning token, and re-sent context byte a single unit of user value consumes. Measured that way, the levers rank differently. Shaving 20 percent off a per-token rate is worth less than removing three steps from an agent loop, and both are worth less than not sending routine requests to a reasoning-class model at all.
A worked example
Numbers make the gap concrete. Consider two features shipped by the same team, calling the same mid-tier model at the same listed price of roughly $1.50 per million tokens blended: a support chatbot answering questions, and a support agent that resolves tickets end to end by planning with a reasoning pass, calling internal tools for order and shipping state, and drafting a resolution.
| Support chatbot | Ticket-resolving agent | |
|---|---|---|
| Model and listed price | Mid-tier, ~$1.50/M blended | Same model, same price |
| Model calls per task | 1 | ~12 steps |
| Context behavior | ~1,500 input tokens, sent once | Starts at ~3,000 tokens, grows ~1,000 per step as tool results accumulate |
| Total input tokens per task | ~1,500 | ~90,000 (context re-sent every step) |
| Output and reasoning tokens | ~400 output | Several thousand, billed at output rates |
| Cost per task | ~$0.003 | ~$0.15 to $0.20 |
| Relative cost | 1x | 50x to 60x |
The pricing page predicts the chatbot well, because that workload looks like the pricing page's implied unit: one call, modest context, no thinking. The agent breaks the prediction entirely, and the driver is the input column: twelve steps of accumulating context sum to roughly 90,000 input tokens before a single answer token is produced. If the planning steps run on a reasoning-class model at frontier rates instead, the multiple moves into the hundreds.
Neither number is wrong, and the agent may well be worth fifty chatbot exchanges if it closes tickets a human would otherwise handle. The point is that the pricing page cannot distinguish the two workloads, and a budget built from per-token rates and request counts will miss by an order of magnitude on the second. Cost per task is the number that survives contact with the workload's actual shape, and it has to be measured, not derived from the rate card.
What actually controls the bill
Since consumption is where modern bills are made, the effective controls are consumption controls, roughly in order of leverage on mixed production traffic.
Match capability to the request. Production traffic is heterogeneous: a large share of requests are routine, and serving them with frontier or reasoning models is pure overspend. Research on the economics of routing, from FrugalGPT to RouteLLM, consistently finds that most traffic can be served by smaller models with little or no quality loss. This is the problem model routing exists to solve per request, and it attacks the largest multiplier first by keeping thinking-class capacity for the requests that need it.
Budget the thinking. Where providers expose reasoning-effort controls or token budgets, set them deliberately per workload rather than accepting defaults. The difference between unbounded and bounded reasoning on routine traffic is often the single largest line item a team can remove in an afternoon.
Engineer for the cache. Prefix caching discounts repeated context heavily, but only when the prefix is byte-identical. Stable system prompts, tool definitions ordered consistently, and dynamic content pushed to the end of the prompt turn the cache from a theoretical discount into a real one.
Trim the loop. For agents: summarize or drop stale tool results, cap step counts, and prune history between steps. Because agent input costs compound with accumulated context, context hygiene pays off quadratically rather than linearly.
Tactical detail on several of these appears in our cost optimization strategies guide; the point here is the ranking, which follows from where the tokens actually go.
The bill follows the product
Per-token prices will likely keep falling, and per-task consumption will likely keep rising, because the second is what falling prices are for. Teams that internalize this stop being surprised by the paradox and start managing the factor they control: how many tokens each unit of user value consumes, and what class of model consumes them. The pricing page is the smaller half of that equation.
A useful first step is instrumentation: know your cost per task, per workload, before optimizing anything. The second is putting the capability decision where it can see each request, which is what a routing layer does, so the expensive tokens are spent only where the request earns them.