Analysis: the price of AI keeps falling, so why are the bills going up?

Two statements are simultaneously true and constantly confused for a contradiction: the price of a unit of AI computation has fallen dramatically, and companies are spending more on AI than ever. The gap between them is where the interesting economics live.

What actually got cheaper

Several forces pushed the price per token down at once:

• Hardware generations. Each accelerator generation delivers more throughput per watt and per dollar.
• Serving optimisations. Continuous batching, paged attention for key-value caches, speculative decoding, quantisation. These are engineering wins, not model changes, and they compound.
• Sparse architectures. Mixture-of-experts designs activate only a fraction of parameters per token, so a very large model can cost like a much smaller one to serve.
• Distillation. Small models trained on the outputs of large ones now handle a wide band of routine work at a fraction of the price.
• Competition. Multiple credible providers at every capability tier, with switching costs low enough to be real.

The result is a genuine, order-of-magnitude decline in price for a given level of capability over a few years. Nobody disputes this.

What got more expensive per task

Meanwhile the number of tokens a single user action consumes has grown, for four reasons:

Longer context. Products that once sent a 500-token prompt now attach documents, conversation history, retrieved passages and tool definitions. Ten thousand input tokens per call is unremarkable.

Reasoning models. Models that produce internal deliberation before answering can generate many times more output tokens per request than a direct-answer model. The rate card did not change; the token count did.

Agents. This is the largest multiplier. One instruction — "reconcile these accounts" — becomes fifteen model calls, each carrying the accumulated context. From a billing perspective that is not one request. It is fifteen, and each is longer than the last.

Retries and evaluation. Production systems retry on failure and run evaluation suites continuously. Both are invisible to users and visible on invoices.

Multiply a 10× price decline by a 30× increase in tokens per task and the bill triples while every unit got cheaper. That is the whole puzzle.

What this does to product margins

The pattern shows up most sharply in per-seat software. A seat costs a fixed amount per month. An AI feature attached to that seat costs whatever the user consumes. The distribution of usage is not normal — a small fraction of users generate a large majority of tokens, and they are usually the most engaged and least likely to churn.

Three responses have become common:

1. Usage-linked pricing. Credits, or tiers with explicit limits. Less elegant than flat pricing, and it survives contact with heavy users.
2. Routing. A small model handles the bulk of requests; the expensive model is reserved for hard ones. In most workloads this is the single largest saving available, ahead of every prompt-level optimisation.
3. Caching and batching. Cached prefixes cut the cost of repeated system prompts and documents; batch endpoints roughly halve the price of anything that does not need an immediate answer.

The infrastructure floor

There is a limit to how far this can fall, and it is physical. Serving a model requires accelerators, memory bandwidth, a building, cooling and a grid connection — and in several major markets, the grid connection is now the binding constraint, with interconnection queues measured in years.

Capital costs are being amortised over volume that has to keep growing to justify it. That works while demand grows. It is also why capacity announcements now read like utility planning documents rather than product news.

What to watch

For anyone running an AI product, three numbers matter more than the headline price per million tokens:

• Tokens per completed task, tracked over time. This captures the thing that is actually growing.
• Share of traffic served by the cheapest adequate model. If it is below half, there is money on the table.
• Cache hit rate on the stable prefix. A byte-level change at the top of a prompt can silently disable caching and raise costs overnight.

The unit price will keep falling. Whether your bill follows depends almost entirely on whether your token count per task holds still — and in a product roadmap full of agents and longer context, it will not do that by accident.