Prefill, Decode, and the KV Cache

Suppose someone asks you:

You’re serving a Llama-2-7B-style model in FP16, with batch size 32 and a 4k context. Roughly how does GPU memory split between model weights and KV cache?

Could you work it out without looking it up? A little while ago, I couldn’t. I knew what a KV cache was, and I knew what the Prefill and Decode phases were, but I did not know how these phases affected compute and memory differently in an actual inference server.

I’ll work through that over this post and the next. This one covers memory, where the KV cache turns out to be the bigger bill. The next covers compute and bandwidth, and why a GPU with enormous compute can still generate tokens slowly.

In my previous post, we stopped after generating one token: “The capital of France is” became Paris. So the question now is, what happens when the model needs to generate the next one? That takes us to three ideas: Prefill, Decode and the KV cache, and eventually back to that GPU-memory question.

Prefer to watch the numbers build? → Open the interactive version

LAB 04 / PREFILL · DECODE · KV CACHE

How much GPU memory does a 7B LLM need?

Watch the number grow one factor at a time, then change the inputs.

Skip to calculator ↓

Theoretical capacity arithmetic · excludes activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation

01 / THE ANSWER

For a 7B-style FP16 model with a 4k context and 32 active sequences: approximately 14 GB of weights and 69 GB of KV cache. Here is how it adds up.

Loading interactive diagram…

With this explainer focused: ← / → step · Space play/pause · R restart.

Read the complete text alternative
  1. The answer. For a 7B-style FP16 model with a 4k context and 32 active sequences: approximately 14 GB of weights and 69 GB of KV cache. Here is how it adds up.
  2. Why cache?. Paris becomes the next input and still crosses every transformer layer. Earlier tokens keep their layer-specific K and V, so the model does not recompute that earlier state.
  3. One number. Start with one number stored in FP16: two bytes.
  4. One Key. One Key contains 32 slices of 128 FP16 numbers: 4,096 numbers, or exactly 8 KiB.
  5. Key + Value. Store the matching Value too. Two 8 KiB vectors make 16 KiB per layer.
  6. Every layer. Every transformer layer keeps its own notes. Across 32 layers, one processed token adds exactly 512 KiB to the cache.
  7. Every token. One set of notes per token. At 4,096 tokens, one active sequence uses 2 GiB, or approximately 2.15 GB.
  8. Every user. Thirty-two people chatting at once means 32 active sequences. Their KV cache reaches 64 GiB, or approximately 68.7 GB.
  9. The model. The model block has not grown. Seven billion FP16 parameters use approximately 14 GB, about 13 GiB.
  10. Compare. The KV cache is about five times the model weights. Together they require approximately 83 GB of theoretical capacity before serving overhead.
  11. Prefill / Decode. Prefill processes the whole prompt and builds the notes. Decode processes one new token at a time, reuses the notes, and extends them. The cache saves compute and costs memory.
KV-cache capacity for the simplified architecture
ContextBatch 1Batch 8Batch 32
1k≈0.54 GB≈4.29 GB≈17.2 GB
4k≈2.15 GB≈17.2 GB≈68.7 GB · tight
8k≈4.29 GB≈34.4 GB≈137 GB · exceeds
16k≈8.59 GB≈68.7 GB · tight≈275 GB · exceeds
32k≈17.2 GB≈137 GB · exceeds≈550 GB · exceeds

Theoretical capacity arithmetic · excludes activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation.

The 32k preset is hypothetical scaling for this architecture. Stock Llama 2 7B uses a 4k context.

After generating Paris, our sequence looks like this:

The capital of France is Paris

The naive approach would be to send all six tokens through the Transformer again to predict the next token. Then all seven for the one after that, then eight, and so on.

If you remember from my previous post, Attention works with three things:

  • Q — Query: What am I looking for?
  • K — Key: When might I be relevant?
  • V — Value: What information can I contribute?

Here is the important realization. Once an earlier token has passed through a Transformer layer, its K and V at that layer don’t change. The Key for “capital” is the same whether we’re generating token 6 or token 100, and so is its Value. Sending all the earlier tokens through the model again would mean recalculating them, again and again.

Enter caching. We store these K and V values in memory instead of throwing them away, so we don’t need to recalculate them. That is the KV cache.

📝 Think of reading a geometry book. To follow chapter 10, you don’t reread chapters 1 to 9. You keep notes as you go and glance back at them. The KV cache is the model’s notes: for every token it has read, a note on when that token might be relevant (K) and what it has to offer (V).


The KV cache

As the model processes tokens, it stores the K and V vectors for every token at every Transformer layer. When a new token arrives, the model attends to those stored vectors instead of calculating them again.

Let’s go back to our example. The model processes “The capital of France is” and predicts Paris. To generate the next token, it now has:

cache already contains K/V for:
The, capital, of, France, is

new input = Paris
↓
at every Transformer layer:
compute Paris's K/V,
attend over the earlier K/V and Paris's current K/V,
then retain Paris's K/V in the cache
↓
predict "."

Then "." becomes the next input, and the cycle repeats.

So after the prompt has been processed, we no longer push the whole sentence through the model as fresh input. One new token enters, travels through every Transformer layer, one token comes out, and one new set of K/V values is retained at each layer.

If you want to carry one concept from this section, it is this:

“The new token travels through the model. Everything before it survives as notes in the KV cache.”


Prefill and Decode

This naturally splits inference into two phases.

When you first send “The capital of France is”, the model has no cached K/V for those tokens yet, so it processes the whole prompt and creates them. This phase is called Prefill:

PROMPT
↓
[The] [capital] [of] [France] [is]
↓
processed through the model
↓
KV cache populated
↓
first generated token: Paris

After that, generation enters the second phase, Decode, where the model works one new token at a time:

new input: Paris
↓
every Transformer layer
↔ attend over the existing cache
+ retain Paris's K/V
↓
predict "."

Then "." becomes the next input, and the cycle repeats.

✏️ Extending the geometry-book example, Prefill is like reading the fundamentals of geometry. Decode is like learning an advanced chapter by referring to those fundamentals, then keeping notes from the new chapter for what comes next.

These sound like two parts of the same operation, but the hardware sees them very differently. Prefill has many tokens available at once, which gives the GPU a lot of computation to do in parallel. Decode has very little new computation per step, yet it has to read the model weights and an ever-growing cache every single time. That difference matters a great deal for GPU utilization, batching and throughput, but I’ll leave that rabbit hole for the next post.

There is a more immediate consequence of the KV cache. It takes memory, and a lot of it.


Back to the original question

Let’s make the memory cost concrete, using a Llama-2-7B-style architecture:

32 Transformer layers
32 KV heads
128 dimensions per head
FP16 = 2 bytes per value

The “heads” line looks more intimidating than it is. I’ll dig into multi-head attention, and how it affects inference, in a later post. For now, think of each Key (K) as 4,096 numbers, which the model splits into 32 slices of 128. The same goes for each Value (V).

For every token, at every layer, we store one Key and one Value:

2              K and V
× 32           KV heads
× 128          dimensions per head
× 2 bytes      FP16 floating-point format

= 16 KiB per token per layer

There are 32 layers in this Llama-2-7B-style model:

16 KiB × 32 = 512 KiB per token

This is the number worth carrying around: about 512 KiB of KV cache for every token. Once you have it, every capacity question is one multiplication away.

Suppose one active sequence reaches 4,096 tokens:

512 KiB × 4,096 tokens = 2 GiB ≈ 2.15 GB

That’s about 2.15 GB of KV cache for one conversation. Now imagine the server handling 32 conversations at once. In serving terms, those are 32 active sequences:

2 GiB × 32 = 64 GiB ≈ 68.7 GB

Meanwhile, the model itself has roughly 7 billion parameters, at 2 bytes each:

7 billion × 2 bytes = 14 billion bytes
                      = 14 GB
                      ≈ 13 GiB

So the answer to the question we started with is:

Model weights    ≈ 14 GB
KV cache          ≈ 69 GB

The KV cache is about five times the size of the model. Together they need about 83 GB of theoretical capacity, leaving almost no practical room on an 80 GB-class GPU once real serving overhead is included.

This is capacity arithmetic, not a measurement. It excludes activations, CUDA graphs, runtime buffers, allocator overhead and fragmentation.


And long context makes it worse

The KV cache grows linearly with the number of tokens it holds. Go from a 4k context to 32k and the cache grows eight times:

512 KiB × 32,768 tokens = 16 GiB ≈ 17.2 GB per sequence
17.2 GB × 32 active sequences ≈ 549.8 GB

This is hypothetical scaling for the same architecture. Stock Llama 2 7B uses a 4k context; the 32k example shows how the capacity would scale if that architecture were extended.

The model hasn’t grown at all. The weights are still about 14 GB. But the cache now needs roughly 550 GB.

If you want to carry one concept from this section, it is this:

“Long context is not only a quality feature. It is a memory-capacity problem.”


The cache is both the solution and the problem

The KV cache exists because recalculating the past would be wasteful. It is what makes generating long responses practical. But the same cache keeps growing as conversations get longer and as more users are served at once, so the thing that saves computation becomes one of the main limits on how much you can serve. Run out of room for the cache, and you can’t admit more conversations, however much compute is sitting idle.

If you want to carry one concept from this post, it is this:

“KV cache memory is a throughput constraint wearing a memory costume.”

There is one thing I deliberately skipped. Why does Prefill tend to keep the GPU’s compute busy, while Decode so often waits on memory bandwidth? That takes us into arithmetic intensity, and into why an enormous amount of compute doesn’t automatically mean generating tokens quickly.

That’s where I’m going next.