Why evicting 90% of the KV cache frees so little memory
Here is a question I could not stop thinking about: if you evict 90% of an LLM’s KV cache, how much memory do you get back? The intuitive answer is “about 90%”. The real answer can be close to zero. Working out why turned into a small tour of how inference engines manage GPU memory, so I wrote it down.
The cache only grows
When a transformer generates text, every new token attends to all the tokens before it. To avoid recomputing keys and values for the whole history at every step, the model stores them, and that store is the KV cache. Each generated token appends its key and value vectors at every layer, and nothing is released until the sequence ends.
It adds up quickly. Take Qwen3-32B: 64 layers, 8 key–value heads (it uses grouped-query attention), and a head dimension of 128. In 16-bit precision, one token costs:
2 (K and V) × 64 layers × 8 heads × 128 dims × 2 bytes = 256 KB per token
Quantise the weights to 4 bits and they still take about 17 GB. On a 24 GB GPU that leaves roughly 6 GB for the cache, which is around 24K tokens. A 32K-token chain-of-thought trace simply does not fit.
The obvious fix: keep only the tokens that matter
Attention is sparse. At any step most of the weight lands on a small set of tokens, and many cached tokens are barely looked at again. Methods such as H2O and SnapKV use this: score every cached token by how much attention it receives, keep the important ones, and evict the rest. On paper, evicting 90% of the tokens should cut cache memory by 90%.
In practice it often does not, for two reasons.
Problem 1: memory is handed out in blocks
Serving engines such as vLLM do not store the KV cache as one long array. They use paged attention: GPU memory is split into fixed-size physical blocks, each holding the keys and values for a small group of tokens (16 by default in vLLM), much like pages in an operating system. A sequence owns a list of blocks, and a block goes back to the allocator only when every slot in it is empty.
That is the catch. Eviction picks tokens by importance, and important tokens are scattered across the whole sequence. Evict most of them and almost every block still holds a survivor or two. The memory is mostly empty, but none of it is free.
The numbers at scale are just as bad. Evict 14K of 16K tokens spread over 1,000 blocks, and even if the survivors were scattered completely at random, a block would be fully empty only about one time in eight (0.87516 ≈ 0.12). You evicted 87.5% of the tokens and got back roughly 12% of the memory. Real importance scores are rarely that kind, because important tokens tend to be spread across the whole sequence.
Why not just reuse the holes?
You could write new tokens into the empty slots instead of freeing whole blocks. The trouble is that the cache stops being in token order. Say token 16,001 lands in the slot that token 40 used to hold: the cache now reads 38, 39, 16,001, 41, and so on. Attention can still compute the right answer, but only if every slot also records its true position, which means extra bookkeeping on every read and a kernel that understands it. It is the same trade-off as heap fragmentation in a memory allocator: filling holes is cheap at first and messy later.
Problem 2: fast attention never keeps the scores
Most eviction methods rank tokens by their attention scores. But FlashAttention, the kernel nearly everyone uses for speed, never stores the full attention matrix. It computes attention tile by tile in fast on-chip memory and throws the scores away, and not materialising that huge matrix is exactly why it is fast. Getting the scores back means falling back to slower “eager” attention, which gives up much of the speed you were trying to protect.
What TriAttention does
TriAttention (Mao et al., 2026) sidesteps the score problem entirely. It never looks at attention scores. Instead, it scores tokens from the geometry of the query and key vectors before rotary position embeddings (RoPE) are applied. In that pre-RoPE space, queries and keys cluster around stable centres, which makes a query predictably favour keys at certain distances, so a token’s importance can be estimated without running attention at all.
For the memory side, the answer is compaction: every so often, the surviving tokens slide forward to close the gaps eviction left behind, whole blocks empty out and go back to the allocator, and the cache stays in token order. That is the last step of the animation above.
On AIME25 with 32K-token generation, the paper reports matching full-attention accuracy with either about 2.5× higher throughput or about 10.7× less KV-cache memory.
The takeaway
KV-cache compression is as much a systems problem as a modelling one. The number that decides whether it helps is not how many tokens you evicted, but how many blocks you gave back. Working on backend systems, this felt familiar: the hard part is rarely deciding what to delete; it is making sure that deleting it actually frees something.