Why did my bill grow faster than my usage?

Most likely conversation length. If each turn re-sends the full history, a 20-turn conversation costs far more than 20 single questions. Summarising or truncating old turns fixes it.

What is prompt caching and how much does it save?

Providers can cache the unchanged prefix of a request — system prompt, tool definitions, a long document — and charge a fraction of the normal input rate on subsequent hits. It only works if the prefix is byte-identical, so stable content must come first.

Is a batch API worth the delay?

For anything not interactive — classification, enrichment, evaluation runs, translation backlogs — yes. Batch endpoints typically cost around half the synchronous rate in exchange for results within hours rather than seconds.

Are reasoning models more expensive than their price list suggests?

Often, yes. They generate internal reasoning tokens before the visible answer, and those tokens are billed as output. The listed rate is per token; the token count per request is what changes.

Business & funding

How AI model pricing actually works: tokens, caching and batching

AI APIs bill by the token, in and out, at different rates — and the levers that cut a bill by 80% are rarely the ones teams reach for first. A practical guide to the cost model.

By DigitalNeuron DeskAug 8, 2026Last updated Aug 22, 20263 min read

Quick answer

How is AI API pricing calculated?

Almost every AI API bills per million tokens, with separate prices for input and output. Output usually costs several times more than input. Cached input, batch processing and smaller models can each cut the bill substantially, and the total for a conversation grows with history because most APIs re-send the whole thread every turn.

Key takeaways

Input and output are priced separately, and output is the expensive side — often three to five times input.
Chat costs grow quadratically with conversation length, because the whole history is re-sent each turn.
Prompt caching and batch endpoints are the two largest discounts available without changing model.
Model routing — small model first, escalate on difficulty — usually beats every prompt-level optimisation.

AI pricing looks simple — a number per million tokens — and then the first invoice arrives and nobody can explain it. The pricing model is not complicated, but it has a shape that punishes the obvious way of building things.

The base unit

Every major provider bills per million tokens, with two separate rates:

Input tokens — everything you send: system prompt, tool definitions, conversation history, attached documents.
Output tokens — everything the model generates.

Output is the expensive side, commonly three to five times the input rate. The reason is mechanical: input can be processed in parallel across the sequence, while output must be produced one token at a time, each requiring a full pass through the model.

The first practical consequence: a verbose answer costs more than a long question. "Summarise in three bullet points" is a cost control, not just a style preference.

Where the money actually goes

Conversation history. This is the one that surprises teams. Most chat APIs are stateless: to continue a conversation you re-send it. Turn 1 sends 500 tokens. Turn 10 sends everything from turns 1–9 plus the new message. Across a long thread the cumulative input grows roughly with the square of the number of turns.

Fixed overhead per request. A 2,000-token system prompt and 3,000 tokens of tool schemas cost 5,000 input tokens on every single call, including the ones where the user typed "thanks".

Reasoning tokens. Models that think before answering emit internal tokens that are billed as output. A request that returns a three-line answer may have generated far more than three lines' worth of tokens.

Retries and agent loops. An agent that takes fifteen steps is fifteen billable requests, each carrying the full context. Retry logic multiplies this quietly.

The four levers, in order of effect

1. Route to a smaller model. The price gap between a frontier model and a small one is typically 10–30×. Most production workloads contain a large fraction of easy requests — classification, extraction, formatting — that a small model handles at full quality. Send those to the small model and escalate only on difficulty or low confidence. Nothing else on this list comes close to the same saving.

2. Cache the stable prefix. Prompt caching charges a small fraction of the input rate for a prefix the provider has already seen. It requires byte-identical prefixes, which imposes a discipline:

[system prompt]        ← stable, cache these
[tool definitions]     ← stable
[reference documents]  ← stable per session
[conversation history] ← changes
[current message]      ← changes

Putting a timestamp or a user name at the top of the prompt defeats caching entirely. This is a common and expensive mistake.

3. Batch what is not interactive. Batch endpoints trade latency for roughly half price. Overnight classification, backfilling embeddings, evaluation suites, bulk translation — none of these need a synchronous response.

4. Shorten the loop. Cap conversation history at N recent turns plus a summary. Trim tool definitions to the ones this task needs. Set explicit output length limits. Individually small; together often a third of the bill.

Costs that are not tokens

Embeddings are cheap per call and easy to run up during re-indexing. Re-embedding an entire corpus after a chunking change is a real line item.
Fine-tuning has a training cost plus, in some cases, a higher inference rate or a hosting charge for the custom model.
Image and audio input convert to token equivalents at rates that vary by provider and resolution — check the specific formula rather than assuming.
Rate limit tiers. Higher throughput sometimes requires committed spend. That is a procurement question, not an engineering one, and it is worth asking early.

Building a forecast you can defend

Instrument first. Log, per request: model, input tokens, output tokens, cached tokens, feature, and user. Without that breakdown, cost optimisation is guesswork.

Then the arithmetic is straightforward:

monthly cost ≈ requests/month
             × (input_tokens × input_rate
              + cached_tokens × cache_rate
              + output_tokens × output_rate)

Model three scenarios — expected, double, and ten times — and check which one breaks your unit economics. A per-seat product with an unbounded chat feature can invert its own margin, and the time to discover that is before launch.

The uncomfortable truth in most reviews is that the biggest saving is not a clever prompt. It is noticing that 70% of traffic never needed the expensive model.

Frequently asked questions

Why did my bill grow faster than my usage?: Most likely conversation length. If each turn re-sends the full history, a 20-turn conversation costs far more than 20 single questions. Summarising or truncating old turns fixes it.
What is prompt caching and how much does it save?: Providers can cache the unchanged prefix of a request — system prompt, tool definitions, a long document — and charge a fraction of the normal input rate on subsequent hits. It only works if the prefix is byte-identical, so stable content must come first.
Is a batch API worth the delay?: For anything not interactive — classification, enrichment, evaluation runs, translation backlogs — yes. Batch endpoints typically cost around half the synchronous rate in exchange for results within hours rather than seconds.
Are reasoning models more expensive than their price list suggests?: Often, yes. They generate internal reasoning tokens before the visible answer, and those tokens are billed as output. The listed rate is per token; the token count per request is what changes.

Sources

API pricing — Anthropic
API pricing — OpenAI
Gemini API pricing — Google

Tagspricingtokenscostcachingbatch