DeepSeek Sparse Attention: How a model reads only the tokens that matter
DeepSeek Sparse Attention lets a language model look at only the most relevant 2,048 earlier tokens for each new word, instead of all of them. Here is how it works and where it is used.

DeepSeek Sparse Attention (DSA) is a way for a language model to read a very long text without comparing every word with every other word. A small, fast scorer picks the most relevant earlier tokens for each new token. The full attention step then runs only on that short list.
The idea matters because the cost of reading long inputs is what limits long-context models.
Skimming a long report with sticky notes
Imagine answering a question about a 500-page report. You could reread every page for every sentence you write. Or you could skim first, flag the 20 pages that look relevant, and read only those closely.
DSA works like the second approach. The skim is cheap and approximate. The close reading is expensive and accurate. Splitting the work this way saves effort when the report is very long.
In a model, a token is a small chunk of text, often a word or part of a word. Attention is the step in which each new token looks back at earlier tokens to decide which ones matter for what it writes next.
Why ordinary attention gets expensive

In standard (dense) attention, every token is compared with every earlier token. Double the length of the input and the number of comparisons roughly quadruples. Engineers write this as O(L^2), where L is the length in tokens.
At 1,000 tokens this is trivial. At 128,000 or 1,000,000 tokens it dominates the bill for compute and memory. That is why many labs look for ways to skip comparisons that are unlikely to matter. Such methods are called sparse attention.
The lightning indexer and the top-k cut
DSA has two parts, according to DeepSeek’s technical report. The first is the lightning indexer. For each new token, it gives every earlier token a relevance score. The report describes the score as a sum over a few small indexer heads, each applying a ReLU function, chosen for speed. The indexer can run in FP8, a low-precision number format.
The second part is fine-grained token selection. It keeps only the key-value entries with the top scores, and attention runs on that set alone. In DeepSeek’s sparse training stage the model selects 2,048 key-value tokens for each query token.
That fixed budget is the point. Whether the prompt has 20,000 or 128,000 tokens, each new token attends to at most 2,048 of them. For a 128K-token prompt, that is roughly 1.6 percent of the context. The report states that the core attention cost falls from O(L^2) to O(Lk), where k is the number of selected tokens.
Where it came from, and how it was added to a trained model
DeepSeek-AI first shipped DSA in an experimental model, DeepSeek-V3.2-Exp. Its GitHub page calls the release an intermediate step toward the next-generation architecture and describes DSA as fine-grained sparse attention for the first time. The company then described it in the DeepSeek-V3.2 report, posted to arXiv on 2 December 2025.
DSA was not trained from scratch. DeepSeek started from its existing DeepSeek-V3.1-Terminus model, which has a 128K context, and added DSA through continued training. The report names two stages:
- Dense warm-up: attention stays dense and only the indexer is trained, to imitate the model’s real attention pattern. The report gives 1,000 steps and 2.1 billion tokens.
- Sparse training: the top-k selection is switched on and all parameters are trained. The report gives 15,000 steps and 943.7 billion tokens.
DSA is built on multi-head latent attention (MLA), DeepSeek’s design that stores a compressed version of each token’s key-value data. The report says DSA uses the MQA mode of MLA, in which each compressed entry is shared across all query heads, for kernel efficiency.
What DeepSeek reports, and who else uses the idea
On quality, the report says DeepSeek-V3.2-Exp showed no substantial performance degradation against V3.1-Terminus on short and long-context tasks. The GitHub page lists close scores, for example 40.1 against 38.5 on BrowseComp and 67.8 against 68.4 on SWE Verified. These are DeepSeek’s own results. On cost, the report describes a significant end-to-end speedup in long-context settings, shown in a figure rather than as numbers in the text.
The technique is now appearing in other labs’ models. Tencent’s new Hy4-base model card describes a gated version of DeepSeek Sparse Attention with an IndexCache, built on multi-head latent attention. In that model each token selects the top 2,048 positions to attend to, and the context window is 1M tokens. The card does not say more in the parts covered by our report, so the gating and IndexCache details are Tencent’s own design.
Trade-offs and what to watch
Sparse attention is a bet that the skim finds the right tokens. If the indexer misses a relevant passage, the model never sees it, and dense attention would not have this failure. DeepSeek’s training tries to reduce the risk by teaching the indexer to match the dense attention pattern.
The saving is also partial. The report notes that the lightning indexer itself is still O(L^2). It is much cheaper than full attention, but it still scores every earlier token. At very long lengths that remaining cost grows, which is why researchers keep looking for cheaper ways to pick tokens.
There is an engineering cost too. The indexer keeps its own scoring pass, and inference software needs dedicated kernels. DeepSeek’s repository points to FlashMLA for the sparse attention kernels and DeepGEMM for the indexer.
Three things are worth watching:
- Whether independent tests confirm that quality holds at 1M tokens, not only at 128K.
- Whether cheaper selection schemes replace the full scoring pass.
- Whether more open-weight models adopt the same top-k recipe, as Tencent has.
Related reading: hybrid attention architectures, KV cache compression and context window and context rot cover other ways to make long inputs affordable.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…
You may also like

Sparse autoencoders: How researchers split a model’s activations into readable features

DeepSeek’s API now does search, link reading and safety checks

A free DeepSeek model nearly matched OpenAI’s gated cyber model at a tenth of the cost
