WHAT IT SHOWS
Why saved work becomes memory.
Prefill builds layer-specific K/V state. Decode reuses it while each new input token still crosses every transformer layer. The calculator turns that mechanism into capacity.
LAB 04 / PREFILL · DECODE · KV CACHE
This continues How an LLM Generates One Token. Paris has been generated. Now follow it into Decode, derive 512 KiB per token, and scale that state by context length and active sequences.
LAB 04 / PREFILL · DECODE · KV CACHE
Watch the number grow one factor at a time, then change the inputs.
Skip to calculator ↓Theoretical capacity arithmetic · excludes activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation
01 / THE ANSWER
For a 7B-style FP16 model with a 4k context and 32 active sequences: approximately 14 GB of weights and 69 GB of KV cache. Here is how it adds up.
With this explainer focused: ← / → step · Space play/pause · R restart.
EXPLORE THE CAPACITY
Weights stay fixed. Per-request state scales.
80 GB-CLASS GPU REFERENCE · 80 GiB ≈ 85.9 GB
MODEL WEIGHTS≈ 14 GB
KV CACHE≈ 68.7 GB
THEORETICAL TOTAL≈ 82.7 GBtight
The main labels use decimal GB. Exact binary values remain visible where they make the derivation clearer. Every bar is scaled from bytes.
The 32k preset is hypothetical scaling for this architecture. Stock Llama 2 7B uses a 4k context.
2 × 32 × 128 × 2 bytes × 32 layers × 4,096 tokens × 32 users = 64 GiB ≈ 68.7 GB
{
"num_hidden_layers": 32,
"hidden_size": 4096,
"num_attention_heads": 32,
"num_key_value_heads": 32,
"torch_dtype": "float16"
}
Llama-2-7b-hf configuration (Meta licence acceptance required).
| Context | Batch 1 | Batch 8 | Batch 32 |
|---|---|---|---|
| 1k | ≈0.54 GB | ≈4.29 GB | ≈17.2 GB |
| 4k | ≈2.15 GB | ≈17.2 GB | ≈68.7 GB · tight |
| 8k | ≈4.29 GB | ≈34.4 GB | ≈137 GB · exceeds |
| 16k | ≈8.59 GB | ≈68.7 GB · tight | ≈275 GB · exceeds |
| 32k | ≈17.2 GB | ≈137 GB · exceeds | ≈550 GB · exceeds |
Theoretical capacity arithmetic · excludes activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation.
The 32k preset is hypothetical scaling for this architecture. Stock Llama 2 7B uses a 4k context.
WHAT IT SHOWS
Prefill builds layer-specific K/V state. Decode reuses it while each new input token still crosses every transformer layer. The calculator turns that mechanism into capacity.
THE BOUNDARY
The values exclude activations, runtime buffers, CUDA graphs, allocator overhead, and fragmentation. The 32k preset is hypothetical scaling for this architecture.