While learning how LLMs work, I kept coming back to a few questions. If the model only takes numbers, what happens to my text? Why can it process the question together, but generate the answer one token at a time? Does it process the whole question again at every step?
I turned that learning into this 8-page comic. It follows one question about a cat through token IDs, sampling, weights, prefill, decode and KV caching.
One question. The same example all the way through.
Start with the question, follow the steps, then try the challenge on the last page.
How does an LLM turn
my question into an answer?
A Large Language Model, explained with one worked example.
THE QUESTIONS WE WILL ANSWER
What does the model
actually receive?
What comes out of
one prediction?
What is inside
the black box?
Can the question be processed
all together?
How does the answer
grow one token at a time?
Does the same question
go in again each time?
ONE EXAMPLE TO FOLLOW
QUESTION
Where did the cat sit?
EXAMPLE ANSWER
The cat sat on the mat.
We’ll follow the text, the token IDs and the growing answer.
I type words. What does the model see?
Text → tokenizer → token IDs → model
GPT-2 example IDs. · marks a leading space. Other tokenizers use different IDs.
THE QUESTION
THE LEARNER
I type a question.
Where does the answer
actually come from?
IMAGINE A GIANT FUNCTION
INPUT / QUESTION
Where did the cat sit?
Question in. Computation. Next piece out.
EVENTUAL EXAMPLE ANSWER
The cat sat on the mat.
BUT I TYPED WORDS
WHAT I TYPED
Where did the cat sit?
Why are there numbers
going into the model?
THE TOKENIZER MAPS TEXT PIECES TO IDs
Where
·did
·the
·cat
·sit
?
TOKEN IDs ENTERING THE MODEL
[8496,750,262,3797,1650,30]
What comes out for our cat question?
464 becomes “The”. That is the first piece of the example answer.
Illustrative probabilities, not a model run. GPT-2 token IDs; chat formatting omitted.
THE SAME QUESTION GOES IN
QUESTION
Where did the cat sit?
INPUT TOKEN IDs
[8496,750,262,3797,1650,30]
SCORES BECOME NEXT-TOKEN PROBABILITIES
NEXT TOKEN
ID
PROBABILITY
The
60%
A
20%
It
10%
All other tokens
10%
Illustrative distribution
SELECT THE NEXT TOKEN
A selection step chooses
one token from the possibilities.
This step is called sampling.
SELECTED TOKEN ID
TURN THE SELECTED ID BACK INTO TEXT
464 → The
ANSWER SO FAR
The
All that for one piece?
What calculated those probabilities?
Ordinary inference uses the learned weights without changing them.
Weights, probabilities and network drawings are illustrative.
LOOK INSIDE THE BLACK BOX
What's inside the box
that processes these IDs?
OUR QUESTION AS TOKEN IDs
[8496,750,262,3797,1650,30]
LEARNED NUMBERS CALLED WEIGHTS
Weights shape the calculation.
They are also called parameters.
EXAMPLE WEIGHTS
0.12 -0.43 0.08
70B ≈ 70 billion parameters
ONE FORWARD PASS
INPUT TOKEN IDs
[8496,750,262,3797,1650,30]
Input → model calculations → scores
AFTER TURNING SCORES INTO PROBABILITIES
The: 60% A: 20%
It: 10% other: 10%
DID ANSWERING CHANGE THE WEIGHTS?
Now it has answered “The”.
TRAINING
Changes the
weights.
ORDINARY INFERENCE
Uses fixed
weights.
Why can it read faster than it can write?
Prefill: known prompt positions can run in parallel. Decode: choices depend on earlier tokens.
Standard autoregressive Transformers. Runtimes may chunk prompts; optimized decoding adds variations.
DOES IT READ MY QUESTION ONE TOKEN AT A TIME?
OUR SIX-TOKEN QUESTION
Where did the cat sit?
The words have an order.
Must they go in one by one?
PREFILL: THE PROMPT IS ALREADY AVAILABLE
All prompt tokens are already known.
Their positions can be processed
in parallel through the model layers.
ID 8496
Where
ID 750
·did
ID 262
·the
ID 3797
·cat
ID 1650
·sit
ID 30
?
No need to generate the question first.
PARALLEL PROCESSING STILL PRESERVES WORD ORDER
Position information keeps the order.
Causal attention lets each position use
itself and the positions before it.
POS. 1
Where
POS. 2
·did
POS. 3
·the
POS. 4
·cat
POS. 5
·sit
POS. 6
?
DECODE: THE ANSWER IS NOT KNOWN IN ADVANCE
Each chosen token affects
the next-token prediction.
DEPENDENT STEPS
The → The cat → The cat sat
Prefill gives the first-token prediction.
Decode continues the dependent loop.
Watch the answer grow, one token at a time.
The context grows. Next: which earlier work does the engine reuse?
These panels show logical context, not a new user request for each token.
STEP 1 (PREFILL): QUESTION → FIRST TOKEN
464 → The
QUESTION / KEPT
Where did the cat sit?
ANSWER SO FAR
(empty)
Answer IDs: []
ANSWER AFTER THIS STEP
The
STEP 2 (DECODE): QUESTION + ANSWER SO FAR → NEXT TOKEN
3797 → ·cat
QUESTION / KEPT
Where did the cat sit?
ANSWER SO FAR
The
Answer IDs: [464]
ANSWER AFTER THIS STEP
The cat
STEP 3 (DECODE): QUESTION + ANSWER SO FAR → NEXT TOKEN
3332 → ·sat
QUESTION / KEPT
Where did the cat sit?
ANSWER SO FAR
The cat
Answer IDs: [464,3797]
ANSWER AFTER THIS STEP
The cat sat
KEEP APPENDING TOKENS TO THE CONTEXT (DECODE)
4. Select 319 → ·on
The cat sat on
5. Select 262 → ·the
The cat sat on the
6. Select 2603 → ·mat
The cat sat on the mat
7. Select 13 → .
The cat sat on the mat.
An end-of-sequence token or a length limit ends generation.
Am I sending the same question again?
One request. A generation loop. Earlier attention state reused.
Next: try the challenge. A new chat turn is a separate request; context handling varies.
ONE ANSWER DOES NOT REQUIRE REPEATED USER INPUT
You submit the question once.
The serving system runs the loop
that generates the answer tokens.
OUR EXAMPLE
One request → The cat sat on the mat.
WHY DOES THE QUESTION APPEAR IN EVERY STEP?
The question remains part of the
context used for each prediction.
With KV caching, earlier attention
keys and values are saved and reused.
SAVED RUNTIME STATE
This is the KV cache.
CACHED DECODE: PROCESS THE NEW TOKEN, REUSE EARLIER STATE
NEW TOKEN INPUT TO THIS DECODE STEP
464 = The
REUSED ATTENTION STATE
The question’s cached keys and values
NEXT TOKEN SELECTED FOR OUR EXAMPLE
3797 → ·cat
A FOLLOW-UP MESSAGE IS A NEW CHAT TURN
EXAMPLE FOLLOW-UP
What color was the mat?
The app supplies relevant conversation
context for the new turn.
Cached prefixes may be reused.
Context reuse is not model training.
Your turn. What happens next?
Explain what happens and why.
Use the example to explain the next step. No need for a long answer.
The model has generated “The cat”. We are using a KV cache.
ORIGINAL QUESTION
Where did the cat sit?
ANSWER SO FAR
The cat
LAST SELECTED TOKEN
3797 = ·cat
The answer is still being generated.
Think about the next decode step.
WHAT TOKEN GOES INTO THE NEXT DECODE STEP?
Which token ID goes into the next decode step,
and what text does that ID represent?
WHAT INFORMATION CAN THE MODEL REUSE?
What earlier information is available from the cache?
Does the user need to send the question again?
DO THE WEIGHTS NEED TO CHANGE?
Can the model continue this answer using fixed weights?
Explain the difference between inference and training.
Write your answers in the comments.
The last page ends with a challenge. Add your explanation to the post on LinkedIn.