Context windows: Why AI forgets, and what context rot means for memory

A context window is the amount of text an AI model can look at in one go: your question, the earlier conversation, any pasted documents and its own reply. Anything outside the window is invisible to the model. Anything inside it competes for the model’s attention.

That second part is why bigger windows have not solved forgetting.

A desk that can only hold so many pages

Picture a researcher working at a desk. Only the pages on the desk can be read. Add more pages and older ones slide off the edge, or get buried under newer ones.

A chatbot works the same way. It does not remember your last chat the way a person does. Each time it answers, the app sends the model the visible conversation again, and the model reads it from scratch. If a detail from an hour ago is no longer on the desk, the model cannot use it.

Tokens, attention and why length has a cost

Models read in tokens, small chunks of text such as a word or part of a word. A window is measured in tokens, so a 100,000-token window holds roughly a long book chapter or several.

Inside the window, the model uses a mechanism called attention. Each token is compared with the other tokens to decide what is relevant. Attention was introduced in the Transformer architecture, described in the 2017 paper “Attention Is All You Need” by Ashish Vaswani and colleagues. Because every token is compared with many others, a longer input costs more computing power and memory.

Anthropic’s engineering team put it this way in September 2025: models have an “attention budget” that each token draws on.

Context rot: More text, less reliable recall

Researchers have found that a model can accept a long input without using it well. In the 2023 paper “Lost in the Middle,” Nelson Liu of Stanford and colleagues tested multi-document question answering and key-value retrieval. They reported that performance is often highest when the relevant information sits at the beginning or end of the input, and drops significantly when it is in the middle. This held even for models built for long contexts.

In July 2025, Chroma, a vector database company, published a technical report titled “Context Rot” by Kelly Hong, Anton Troynikov and Jeff Huber. It tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 variants. The authors reported that performance grows increasingly unreliable as input length grows, even on simple tasks. Related but wrong passages, called distractors, hurt more as inputs got longer.

One result is easy to relate to. On a chat-memory benchmark called LongMemEval, models did clearly better on a short, focused prompt of about 300 tokens than on a full chat history of about 113,000 tokens. Chroma sells retrieval tools, so readers may weigh its framing accordingly, but the effect is consistent with the earlier academic work. Anthropic uses the term context rot for the same decline in recall.

Working around the limit: Summaries, retrieval and memory

Since a window is finite and long inputs degrade, builders curate what goes in. Anthropic calls this context engineering and describes the context as a finite resource with diminishing marginal returns. Common techniques include:

• Compaction: when a conversation nears the limit, summarize it and start a new window with the summary.
• Clearing old tool results: Anthropic calls this one of the safest, lightest-touch forms of compaction.
• Retrieval: store material outside the window and pull in only the relevant pieces when needed.
• Memory features: keep notes between sessions so that facts survive when the window is cleared.

Each trade-off is real. A summary can drop a detail that later matters. Retrieval can fetch the wrong passage. Memory can keep something that is no longer true.

Where it shows up today: Meta’s Active Memory

The problem is not only theoretical. In our report on Meta’s specialized Muse and Meta AI models, the company says its Active Memory feature trains models to separate short-lived information from lasting facts, spot stale details, and recognize when something said earlier becomes relevant again. Those are the same weaknesses that long contexts expose: old facts pile up, some go stale, and the useful one gets buried.

Meta reports that on its own Active Memory evaluation, 90% of tasks correctly use critical context. That figure comes from Meta’s evaluations, and the test sets are not published, so it cannot be compared with the independent studies above.

What to watch

Window sizes keep growing, but the research suggests size alone is not the measure. Useful questions are how a model performs when the answer is buried in the middle, how it handles distractors, and whether a product’s memory forgets stale facts rather than only storing more. Independent long-context tests, and published test sets from vendors, will show whether tricks like trained memory behavior beat simply adding more tokens.

For everyday use, the practical lesson is simple. A short, focused prompt often works better than a long dump of text. Starting a fresh chat with a clean summary can beat continuing a very long one.