Long context
Put the whole document set into the window and ask questions over it.
- No index to build or keep fresh
- Sees cross-document links a search might miss
- Strong at finding one fact
- Every question pays for every token
Guide · Context engineering
What is a context window? The model’s working memory, counted in tokens.
Everything a model can consider when it writes a reply — instructions, history, documents, tools and the reply itself — has to fit in one window.
A language model has no memory between requests. Each call sends it a block of tokens, and the context window is the most that block can hold. This guide explains what a token is, what counts toward the window, how big windows have become, why models still miss facts placed in the middle of long inputs, and how to choose between loading everything and retrieving only what a step needs.
Updated · 9 sources
What counts toward one request’s window
01
System promptrole, rules and output format
input02
Tool definitionseach tool’s name, purpose and schema
input03
Conversation so farearlier messages and tool results
input04
Documents and imagesfiles, pages and images in the request
input05
The new messagewhat the person just asked
input06
The replythe output, reasoning included
outputWhat is a context window in AI?
A context window is the amount of text, measured in tokens, that a language model can take into account at once — the input it is given plus the output it writes. Claude’s documentation calls it the model’s “working memory”, distinct from the data it was trained on. Anything outside the context window is invisible to the model on that call.
The unit of measure
Tokens: how a context window is measured.
Models read tokens — word pieces, punctuation, and fixed-size patches of images and audio. Each model family has its own tokenizer, so the same text is a slightly different count on each.
Counting before you send
- Count endpointClaude’s
count_tokenstakes the same input as a real request — system prompt, tools, images, PDFs — and returns an estimate. - Usage on every replyEach response reports the tokens the request consumed, cached or not.
- Caching counts tooA cached prefix is cheaper, but it still occupies the window.
Output shares the window. The reply is written into the same window as the input. A request that fills the window with input leaves no room for the answer — Claude returns a “prompt is too long” error when the input alone is over.
How big they have become
How context window sizes grew from 2,048 tokens to a million.
In under four years — May 2020 to February 2024 — the largest windows grew about 490-fold (1,000,000 ÷ 2,048, our arithmetic). What fits changed from a few pages to a code base — but, as the next section shows, what a model can reliably use did not grow at the same rate.
- May 20202,048 tokensGPT-3’s context window: a few pages of text, enough for few-shot examples.
- May 2023100,000 tokensAnthropic expands Claude from 9K to 100K tokens — about 75,000 words.
- Feb 20241 million tokensGoogle announces Gemini 1.5 with a window of up to 1 million tokens for early testers.
- Oct 20261M as the defaultAs of October 2026, current Claude models have a 1M-token window by default, with no beta header; older models keep 200K.
Where recall drops
Lost in the middle: why a full context window recalls less.
In 2023 Liu and colleagues moved one relevant document through a stack of irrelevant ones and measured accuracy at each position. The result was a U-shaped curve: models used facts at the start and end of the input well, and facts in the middle badly.
Start of the inputEnd of the input
123
- BeginningHigh accuracy — the paper calls it primacy bias.
- MiddleLowest accuracy. With 20 or 30 documents, GPT-3.5-Turbo scored below its no-documents baseline.
- EndHigh accuracy again — recency bias, close to the question.
What helps. Put the material that matters at the start or right beside the question. The paper found that repeating the question before and after the data made a simple lookup task near-perfect — though it changed little on harder question answering. Past about 20 retrieved documents, adding more barely moved accuracy.
Load everything, or search
Long context vs retrieval: when to load everything into the context window.
A million-token window tempts teams to skip retrieval and send everything. Sometimes that is right. The trade-off is cost per question, freshness, permissions and recall on many facts at once.
Retrieval
Search a store for the passages this question needs and send only those.
- Scales to corpora far beyond any window
- Small, cheap, fast requests
- Filters by who is asking before the model reads
- Only as good as the search
Every token, every time (caching helps)Cost per questionOnly the passages found
Bounded by the windowCorpus sizeBounded by the index
Weaker — Google notes accuracy drops when several facts are sought at onceMany facts at onceEach fact searched for directly
Resend after every changeFreshnessRe-index the changed document
Everyone sent the same bundlePermissionsResults filtered per person
How large is the material, and how often is it asked about?
Fits the window, asked onceSend it all. Reading one contract or one report end to end is what long context is for.
Fits the window, asked oftenSend it all and cache the stable prefix, or retrieve — compare the cost per question.
Larger than any windowRetrieve. Search for the passages each question needs, filtered by who is asking.
In long conversations
Managing a context window across a long conversation.
Each turn adds the person’s message, the reply and every tool result. Left alone, the window fills; managed, an agent can work far beyond one window’s worth of tokens.
- Count
Estimate tokens before each call; read the usage after it.
- Budget
Give instructions, history, retrieval and tools each an allowance.
- Order
Stable blocks first so they cache; key facts near the question.
- Clear
Drop old tool results once they have been used.
- Compact
Near the limit, summarise and continue in a fresh window.
- Note
Write what must survive to a store outside the window.
On Botminds
How Botminds manages each agent’s context window.
On Botminds the context window is a budget the platform manages for every agent on every turn — split into tiers, filled by priority, and open to inspection before a model reads it.
A managed context windowbudgeted, ordered and previewable, every turn
Tiers with budgetseach kind of context gets its own allowance
Priority decidesover budget, the lowest priority goes first
Every drop recordeda strange answer traces to what the model did not see
A note in its placeoptionally, a line telling the agent what was dropped
Stable firstunchanging blocks lead, so they can be cached
Folding memorylatest messages word for word, one rolling summary
Names firstone line per skill or table; full bodies on demand
Preview with token countsevery block and its tier, with no model call
Read how the budget works in context engineering on Botminds, and how an agent’s model, instructions and tools fit together in the anatomy of an agent.
Questions people ask
What is a context window: common questions.
The technology behind itContext engineeringExactly what the model sees, each turn→What is a context window in an LLM?
A context window is the number of tokens a language model can consider at once: the system prompt, the conversation, any documents and tool results, and the reply it writes. Text outside the context window is invisible to the model on that call.
What happens when the context window is full?
If the input alone is too long, the request is refused — Claude returns a “prompt is too long” error. If the reply runs into the end of the window, generation stops. Chat apps avoid both by dropping, summarising or compacting older turns.
How many words is 1 million tokens?
Using Google’s rule of thumb of 60–80 English words per 100 tokens, a million tokens is roughly 600,000 to 800,000 words — Google’s examples are eight average-length novels or 50,000 lines of code.
Is a bigger context window always better?
No. A bigger window lets more fit, but recall falls as inputs grow — Claude’s documentation calls this context rot, and the Lost in the Middle paper showed facts in the middle of long inputs are used worst. Choosing what goes in still matters.
What is the “lost in the middle” problem?
It is the finding, from Liu et al. (2023), that language models use information at the beginning and end of a long input much better than information in the middle, giving a U-shaped accuracy curve as the relevant passage moves through the input.
Does the output count toward the context window?
Yes. The reply is written into the same window as the input, and on reasoning models the thinking tokens count too. Leave room for the answer when you fill the window.
Should I use long context or RAG?
Use long context when the material fits and is read once or a few times. Use retrieval when the corpus is larger than any window, changes often, or must be filtered by who is asking — and to keep the cost of each question low.
Do images and audio use the context window?
Yes. They are converted to tokens too. On Gemini, a small image costs 258 tokens and audio about 32 tokens per second; other providers count differently, so use the provider’s token counter.
Sources
- Claude documentation: Context windowsplatform.claude.com
- Claude documentation: Token countingplatform.claude.com
- Google Gemini API: Understand and count tokensai.google.dev
- Google Gemini API: Long contextai.google.dev
- Liu et al.: Lost in the Middle — How Language Models Use Long Contexts (TACL)arxiv.org
- Chroma: Context Rot — How Increasing Input Tokens Impacts LLM Performance (2025)trychroma.com
- Brown et al.: Language Models are Few-Shot Learners (GPT-3, 2020)arxiv.org
- Anthropic: Introducing 100K Context Windows (May 2023)anthropic.com
- Google: Our next-generation model, Gemini 1.5 (February 2024)blog.google
Read next
Context engineering vs prompt engineeringWording one request, or deciding everything the model sees→AI agent memory typesWorking, episodic, semantic and procedural memory, and how agents use them→Agentic RAG vs RAGHow retrieval changes when an agent decides when, where and how to search→All guidesEvery explainer, by topic→
See it on
your own work.
Bring one process and its documents. We’ll show you the agents, the controls and the record, running on your data.