My initial attempts to learn how LLMs work were not very productive. Disparate, too deep or too shallow and sometimes even outdated information slowed me down until I bumped into 3Blue1Brown on YouTube. That gave me a direction, although I found it still too dense and heavy for non-mathematically leaning folks.
So this is my attempt to share my “distilled” understanding of how LLMs work without getting into the nitty-gritties of mathematical or ML details. The question I will try to answer is:
What actually happens when an LLM generates one token?
LAB 03 / ANATOMY OF A TOKEN
From a prompt to Paris.
Follow one prediction through the model, one idea at a time.
Conceptual illustration · no model is running
01 / PROMPT
Let’s follow what happens before the model generates the next token.
With this explainer focused: ← / → step · Space play/pause · R restart.
Read the full text explanation
- Prompt. Let’s follow what happens before the model generates the next token.
- Tokens & embeddings. For simplicity, we’ll treat each word as one token. Each token ID selects a learned embedding. Meaning is represented as a pattern distributed across many dimensions, not one concept per cell.
- Transformer. Token representations pass through transformer layers. Attention and MLP are the two main operations; normalization, positional information, and residual connections also play a part.
- Q / K / V. Query: what am I looking for? Key: when might I be relevant? Value: what information can I contribute? These are analogies for learned vectors, recalculated at every layer.
- Attention. Q and K determine relevance. V supplies information. France can gather information from itself and earlier tokens, including capital, to become more context-aware. The strengths and feature descriptions here are illustrative.
- MLP & layers. Attention gathers information across allowed token positions. MLP transforms the current token’s representation independently, using patterns learned during training. Both repeat across many layers, with fresh Q, K and V at each layer.
- Prediction. By the final layers, “is” carries enough context to support the next-token prediction. Output projection produces logits; softmax converts them to probabilities. The example probabilities are invented and include 3% for other tokens.
- One token. This illustrated sample selects Paris; sampling does not always choose the most probable token. To predict the next token, decode reuses earlier K/V while the new token still passes through every transformer layer. Next: prefill, decode, and the KV cache.
Next: Why an LLM doesn’t recompute every previous token. A follow-up is planned.
Let’s start with a concrete example. Let’s say that when you open up your ChatGPT, you type this:
The capital of France is.
And then you get a response saying that the capital of France is Paris, followed by a full stop. So if you notice, two additional tokens are generated. One is “Paris,” and the other is “full stop.”
To understand what happens behind the scene, the first concept to grasp is tokenization. For simplicity, you can assume tokenization means each word becomes a “token” (or tokens) that the model treats as a processable entity.
So in our particular example, the text we typed gets divided into five tokens:
[the, capital, of, France, is]One fundamental thing to remember is that everything an LLM can process has a multi-dimensional vector representation and almost all operations in the LLM involve vectors and matrices.
Why a multi-dimensional vector? Because meaning is not stored in a single number; it is represented as a pattern distributed across many dimensions of the vector.
Just as we learn from a vocabulary when learning a language, an LLM is trained with a fixed vocabulary of tokens. Each token has an ID, and the model uses an embedding matrix to convert that token ID into a learned vector representation that it can process.
So, after tokenization in our example, the first step is that the LLM (or the model architecture) retrieves the embedding for this set of tokens.
If you want to carry one concept from what we have described so far, it is this:
“Input texts are broken down and converted into a mathematical representation for futher operations”
Once we get a set of embeddings/vector representations from the tokens it goes through multiple layers of processing called the “Transformer”. Each transformer layer uses various mathematical operations to progressively enrich the tokens’ representation. After many such layers, the model gains enough contextual information to predict the next token.
In each transformer layer, multiple operations happen, but two are most significant:
- Attention
- MLP or Multilayer Perceptron
But to understand these two operations we need to understand related concepts:
- Q or Query (“What am I looking for?”)
- K or Key (“When might I be relevant?”)
- V or Value (“As per my relevancy what information/features can I contribute?”)
Each token/embedding has these, which you can think of as part of its “personality”. Q, K and V are not fixed. They are recalculated at every transformer layer as the token becomes more context-aware.
For example, the Q for our “France” token (Q_France) could encode the query: “Given that I encode France/country/place-related features, what earlier information is relevant to me?”
These query can be “answered” by all the previous tokens in this sequence [the, capital, of] which would provide additonal context to this “France” token.
For example K of “capital” token could have “I may be relevant to queries involving things like country–capital relationships, capitalization, or related linguistic patterns.” and Q_France reaches their a mathematical op happens that decides the relevance of it and shares the value of “capital” that could be:
- concept of “capital city”
- relationship between a capital and a country/place
- noun/syntactic features
- possibly other meanings associated with “capital”
To the token “France”. This mechanism is called “Attention”.
When the model compares the query for “France” with the key for “capital,” it calculates how relevant “capital” is. If the relevance is high, more of the information represented by the value of “capital” contributes to the representation of “France”.
One concept from what we have described so far is this: the current token would ask:
Attention: What information from earlier tokens matters to me?
The next important mechanism is called MLP or Multilayer Perceptron.
To understand this, let’s go back to our example of vocabulary. For us to speak or write, knowing the vocabulary is not enough, life situations and our education “trains” us to use the facts and vocabulary we learnt in a much more contextual and enriched way so that we can use it in situations we have never seen.
Similarly, just having the vocabulary isn’t enough for the model; it gets trained to gain more concepts about vocabulary and facts, enabling it to apply this knowledge in completely new situations.
Simply speaking, Multilayer Perceptron, is about enriching a token. In our example, the “France” token would ask the model:
MLP: Given what I now know, what useful features can I derive using the patterns learned during training?
There are other operations such as RMSNorm and RoPE happens that further stabilizes the scale and enrich the token with positional data but that for later.
This set of operations repeats multiple times, and each time “France” gains more context for prediction.
Exact same thing happens for the “is” token which is the last token for our input. By the end of the process, the “is” token, intuitively knows that we are looking for capital city of France.
Then model produces a set of possible next tokens with different likelihoods and chooses one.
Skipping a few mathematical steps, you can think of the model as producing something like:
Paris 90%
Marseille 5%
London 2%and then model based on this sampling finally chooses Paris. A few operations happen during these phases (Unembedding, Output projection/ Logit generation, Softmax), which we might explore in more detail later.
But one concept to take from this phase is:
“Based on the context and relationship of the tokens we have seen so far, we predict the most probable token.”
Everything we have described so far gets us to the first generated token: “Paris.”
But generating the next token is slightly different. The model doesn’t want to redo all that work.
And that takes us to three important ideas: prefill, decode, and the KV cache.