LAB 04 / PREFILL · DECODE · KV CACHE

The model stays fixed.
The cache grows.

This continues How an LLM Generates One Token. Paris has been generated. Now follow it into Decode, derive 512 KiB per token, and scale that state by context length and active sequences.

LAB 04 / PREFILL · DECODE · KV CACHE

How much GPU memory does a 7B LLM need?

Watch the number grow one factor at a time, then change the inputs.

Skip to calculator ↓

Theoretical capacity arithmetic · excludes activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation

01 / THE ANSWER

For a 7B-style FP16 model with a 4k context and 32 active sequences: approximately 14 GB of weights and 69 GB of KV cache. Here is how it adds up.

Loading interactive diagram…

With this explainer focused: ← / → step · Space play/pause · R restart.

Read the complete text alternative
  1. The answer. For a 7B-style FP16 model with a 4k context and 32 active sequences: approximately 14 GB of weights and 69 GB of KV cache. Here is how it adds up.
  2. Why cache?. Paris becomes the next input and still crosses every transformer layer. Earlier tokens keep their layer-specific K and V, so the model does not recompute that earlier state.
  3. One number. Start with one number stored in FP16: two bytes.
  4. One Key. One Key contains 32 slices of 128 FP16 numbers: 4,096 numbers, or exactly 8 KiB.
  5. Key + Value. Store the matching Value too. Two 8 KiB vectors make 16 KiB per layer.
  6. Every layer. Every transformer layer keeps its own notes. Across 32 layers, one processed token adds exactly 512 KiB to the cache.
  7. Every token. One set of notes per token. At 4,096 tokens, one active sequence uses 2 GiB, or approximately 2.15 GB.
  8. Every user. Thirty-two people chatting at once means 32 active sequences. Their KV cache reaches 64 GiB, or approximately 68.7 GB.
  9. The model. The model block has not grown. Seven billion FP16 parameters use approximately 14 GB, about 13 GiB.
  10. Compare. The KV cache is about five times the model weights. Together they require approximately 83 GB of theoretical capacity before serving overhead.
  11. Prefill / Decode. Prefill processes the whole prompt and builds the notes. Decode processes one new token at a time, reuses the notes, and extends them. The cache saves compute and costs memory.
KV-cache capacity for the simplified architecture
ContextBatch 1Batch 8Batch 32
1k≈0.54 GB≈4.29 GB≈17.2 GB
4k≈2.15 GB≈17.2 GB≈68.7 GB · tight
8k≈4.29 GB≈34.4 GB≈137 GB · exceeds
16k≈8.59 GB≈68.7 GB · tight≈275 GB · exceeds
32k≈17.2 GB≈137 GB · exceeds≈550 GB · exceeds

Theoretical capacity arithmetic · excludes activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation.

The 32k preset is hypothetical scaling for this architecture. Stock Llama 2 7B uses a 4k context.

Why saved work becomes memory.

Prefill builds layer-specific K/V state. Decode reuses it while each new input token still crosses every transformer layer. The calculator turns that mechanism into capacity.

Theoretical capacity arithmetic.

The values exclude activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation. The 32k preset is hypothetical scaling for this architecture.