What is a context window, and why does it run out?
The context window is everything a model can see at once — your prompt, the conversation, retrieved documents and its own reply. Here is how it is measured, why longer is not automatically better, and what to do when you hit the limit.
Kurze Antwort
What is a context window in an AI model?
A context window is the maximum amount of text, measured in tokens, that a model can consider in a single request. It holds the system instructions, the conversation so far, any documents you paste in, and the answer being generated. When the total exceeds the limit, something has to be dropped or summarised.
Das Wichtigste
- The window is measured in tokens, not words — roughly 0.75 words per token in English, and far fewer characters per token in Korean or Japanese.
- Everything shares the same budget: instructions, history, attached files, tool definitions and the output.
- Models reliably use the beginning and end of a long context better than the middle, so placement matters.
- Cost and latency scale with the tokens you actually send, which is why caching and retrieval usually beat pasting everything in.
Every conversation with a language model has a hard ceiling on how much it can see at once. That ceiling is the context window, and almost every strange behaviour people report — the model "forgetting" what you said, ignoring an attached file, losing the thread halfway through — traces back to it.
Tokens, not words
Models do not read characters or words. Text is first split into tokens: common fragments that the tokeniser has learned. In English, a token averages about four characters, so 1,000 tokens is roughly 750 words.
That ratio is not universal. Scripts that fall outside the tokeniser's training distribution fragment more:
| Text | Approximate tokens |
|---|---|
The quick brown fox (19 chars, English) | ~4 |
안녕하세요 반갑습니다 (11 chars, Korean) | ~10 |
こんにちは、はじめまして (12 chars, Japanese) | ~11 |
| A 4-space indented code line | 1 token per indent level |
The practical consequence: a Korean or Japanese document consumes noticeably more of the window than an English document of the same visible length, and costs proportionally more per request.
What competes for the space
It is tempting to think of the window as "how long a document I can paste". It is really a shared budget:
- The system prompt — the instructions that define the assistant's behaviour.
- Tool definitions, if the model can call tools. A dozen tools with detailed schemas can run to thousands of tokens before you have said anything.
- The conversation history, usually re-sent in full on every turn.
- Attached or retrieved documents.
- The output. Generated tokens come out of the same budget in most APIs, which is why a very long input can leave no room for a long answer.
If you have ever attached a large PDF and received a truncated response, this is why.
Long does not mean uniformly good
Research on long-context behaviour — most influentially Lost in the Middle — found a consistent pattern: models retrieve facts placed near the start or the end of a long input far more reliably than facts buried in the middle. The curve is U-shaped, and it does not fully flatten as windows get larger.
Three rules follow directly:
- Put instructions first and the immediate question last. The two positions the model attends to best.
- Do not pad. Twenty relevant pages beat two hundred pages containing the same twenty.
- Test at realistic length. A prompt that works at 5,000 tokens can quietly degrade at 100,000.
Vendor "needle in a haystack" benchmarks — hiding one sentence in a long document and asking the model to find it — measure the easy case. Retrieving a single distinctive fact is much simpler than reasoning over material spread across the whole input.
Cost, latency, and caching
You pay for tokens you send, on every request. A 100,000-token context re-sent across a 20-turn conversation is two million input tokens, even if the user typed twenty short questions.
Two mechanisms take the edge off:
- Prompt caching. Providers can cache the unchanged prefix of a request — system prompt, tool definitions, a long document — and charge much less for cache hits. It works only if the prefix is byte-identical, so put stable material first and volatile material (timestamps, user names) last.
- Batching. For work that is not interactive, batch APIs typically cost significantly less in exchange for delayed results.
Latency follows a similar shape: time-to-first-token grows with input length, so a chat that feels instant with a short prompt feels sluggish once you attach a large file.
What to do when you hit the limit
Retrieve instead of paste. Index your documents, fetch the handful of passages that bear on the question, and send those. Requests stay small, cheap and accurate. This is the whole argument for retrieval-augmented generation.
Summarise the history. Replace old turns with a compact summary of decisions and facts established so far. Keep the most recent turns verbatim — that is where the immediate thread lives.
Split the task. Two focused requests usually outperform one enormous request. Extraction then analysis; per-document then combine.
Trim your tools. If the model only needs three tools for this task, do not send thirty.
Measure before optimising. Count tokens on a real request. Teams are routinely surprised to find that the system prompt or an unused tool schema, not the user's document, is consuming most of the window.
The context window is not a feature to maximise. It is a budget to spend deliberately.
Häufige Fragen
- How many words fit in a 200,000-token context window?
- Roughly 150,000 English words, or about 500 pages of ordinary prose. Korean, Japanese and Chinese text uses more tokens per character, so expect substantially fewer pages for the same limit.
- Does a bigger context window remove the need for RAG?
- No. A large window makes retrieval less fiddly, but sending 500 pages on every request is slow and expensive, and accuracy still degrades in the middle of very long inputs. Retrieval keeps requests small and targeted.
- What happens when a conversation exceeds the window?
- The application has to intervene — usually by dropping the oldest turns, summarising them, or moving them into a retrieval store. The model itself simply cannot see anything outside the window.
- Why does my long conversation get more expensive over time?
- Most APIs re-send the entire conversation with every turn, so the input grows with each exchange. Prompt caching reduces the cost of the repeated prefix, but the tokens are still being processed.