WHAT IT SHOWS
From text to a prediction.
Tokenization, embedding lookup, attention, and MLP build the context used to predict a next token. Each step can be read at your own pace or played as a continuous lesson.
LAB 03 / HOW AN LLM GENERATES ONE TOKEN
One token.
Many transformations.
What happens between a prompt and a prediction? Follow the representations through the model, then step back to inspect each idea.
LAB 03 / ANATOMY OF A TOKEN
From a prompt to Paris.
Follow one prediction through the model, one idea at a time.
Conceptual illustration · no model is running
01 / PROMPT
Let’s follow what happens before the model generates the next token.
Loading interactive diagram…
With this explainer focused: ← / → step · Space play/pause · R restart.
Read the full text explanation
- Prompt. Let’s follow what happens before the model generates the next token.
- Tokens & embeddings. For simplicity, we’ll treat each word as one token. Each token ID selects a learned embedding. Meaning is represented as a pattern distributed across many dimensions, not one concept per cell.
- Transformer. Token representations pass through transformer layers. Attention and MLP are the two main operations; normalization, positional information, and residual connections also play a part.
- Q / K / V. Query: what am I looking for? Key: when might I be relevant? Value: what information can I contribute? These are analogies for learned vectors, recalculated at every layer.
- Attention. Q and K determine relevance. V supplies information. France can gather information from itself and earlier tokens, including capital, to become more context-aware. The strengths and feature descriptions here are illustrative.
- MLP & layers. Attention gathers information across allowed token positions. MLP transforms the current token’s representation independently, using patterns learned during training. Both repeat across many layers, with fresh Q, K and V at each layer.
- Prediction. By the final layers, “is” carries enough context to support the next-token prediction. Output projection produces logits; softmax converts them to probabilities. The example probabilities are invented and include 3% for other tokens.
- One token. This illustrated sample selects Paris; sampling does not always choose the most probable token. To predict the next token, decode reuses earlier K/V while the new token still passes through every transformer layer. Next: prefill, decode, and the KV cache.
Next: Why an LLM doesn’t recompute every previous token. A follow-up is planned.
THE BOUNDARY
A conceptual illustration.
No model is running here. Token splits, IDs, vector cells, relevance strengths, and probabilities are illustrative. English feature descriptions are analogies for learned patterns in vectors.