KV cache compression: how LLMs shrink the memory that long prompts demand

KV cache compression is a set of techniques that shrink the memory a language model keeps about the text it has already read. That memory, the key-value (KV) cache, grows with every token, so it becomes the main cost of serving long prompts. Compressing it lets the same hardware handle longer contexts and more users at once.

Why every token leaves a memory trail

A transformer model generates text one token at a time. To produce each new token, its attention layers compare the new position with every earlier one. Each earlier token has two stored vectors per layer: a key (a label describing what the token offers) and a value (the content it passes along). Recomputing them at each step would be wasteful, so the model saves them. That saved collection is the KV cache.

An analogy: a student reading a long book keeps an index card for every page. The cards make it quick to look things up, but after a million pages the card box is bigger than the student’s desk. The cache grows linearly with context length and with the number of simultaneous requests, and for long prompts it can exceed the memory taken by the model’s weights. Reading it back for every new token also consumes memory bandwidth, which slows generation.

This is the problem behind the technical report that DeepSeek cites for its latest open model. Our coverage of the DeepSeek-V4.1-Flash-1017 release notes that the card recommends a 1M-token context window and cites a report titled “Pushing the Limits of KV Cache Compression.”

Sharing keys and values across attention heads

The first family of methods changes the architecture so fewer vectors are stored. In standard multi-head attention, every attention head has its own keys and values. In 2019, Noam Shazeer proposed multi-query attention, where all heads share a single set of keys and values. He reported that this cuts the memory traffic of incremental decoding, making it faster, with only minor quality loss.

In 2023, Joshua Ainslie and colleagues introduced grouped-query attention, which sits between the two extremes: heads are split into groups, and each group shares one key-value set. The authors also showed that an existing multi-head model can be converted by further training with about 5% of the original pre-training compute, reaching quality close to multi-head attention at a speed comparable to multi-query attention. Grouped-query attention is now a common design choice in open models.

Squeezing the cache into a smaller latent vector

A second approach compresses what is stored rather than how many heads exist. DeepSeek-AI introduced Multi-head Latent Attention (MLA) in the DeepSeek-V2 paper, submitted in May 2024. Instead of caching full keys and values, the model caches a compact latent vector from which they can be reconstructed when needed. In the paper’s comparison with DeepSeek 67B, the authors report a 93.3% smaller KV cache and up to 5.76 times the maximum generation throughput (their own figures, alongside a 42.5% reduction in training cost).

The idea differs from sharing: each head keeps its own view of the data, but the stored representation is a compressed summary. The trade-off is a more complex design and extra computation to expand the latent vector. DeepSeek has built on this line of work since V2, which is why its later models are often discussed in connection with cache efficiency.

Storing fewer bits, and managing the memory better

A third family leaves the architecture alone and shrinks the numbers. KV cache quantization stores keys and values at lower precision. KIVI, by Zirui Liu and colleagues (arXiv 2024, ICML 2024), quantizes keys per channel and values per token and needs no tuning. The authors report 2-bit storage with almost the same quality on Llama, Falcon and Mistral models, 2.6 times less peak memory including model weights, and 2.35 to 3.47 times higher throughput because larger batches fit.

A different problem is waste. In 2023, Woosuk Kwon and colleagues at UC Berkeley and collaborators published PagedAttention and the vLLM serving system. Because each request’s cache grows unpredictably, older systems reserved big contiguous blocks and wasted much of them. PagedAttention splits the cache into small pages, borrowing the idea of virtual memory from operating systems, and lets requests share pages. The authors report 2 to 4 times higher throughput than systems such as FasterTransformer and Orca at similar latency. This does not compress the data, but it makes the memory that exists far more usable.

Other methods drop or merge tokens the model is unlikely to need again. They save memory without changing precision, but they carry a risk covered below.

The price of forgetting, and what to watch

Every compression method trades something. Sharing heads or using a small latent vector can reduce quality unless the model is trained for it from the start. Low-bit quantization can degrade accuracy on tasks that depend on exact details. Dropping tokens can remove a fact that becomes relevant many steps later, which matters for long documents and agent sessions that revisit earlier output. Results also depend on the model and the task, and most published numbers come from the authors’ own tests, so independent reproduction is worth looking for.

Several things are worth watching. First, how much of a model’s long-context ability survives compression in agent workloads, where a single session may run for hours. Second, whether labs publish the details: in the DeepSeek card covered by our news story, the technical report is linked, but the card gives no exact figures for the time and length savings it describes, and its benchmark results are DeepSeek’s own. Third, how serving software such as vLLM, SGLang and llama.cpp adopts these formats, since a compression scheme only helps readers if it runs in the tools they use. As context windows reach a million tokens, the cache, not the model’s weights, is increasingly what decides what is affordable to run.