§1The autoregressive loop

"Autoregressive" means the model's own output is fed back in as input. One forward pass predicts one token; generation repeats this loop until the model emits the special <eos> (end-of-sequence) token or hits a length cap.

§2Sampling lab

The model received the cat sat on the and produced this distribution over what comes next. Dashed outlines are the original probabilities. Solid bars are what is left after your settings rescale and filter them. Then sample from it a few hundred times and watch which tokens actually get picked.

Try temperature 0.05, where nearly every draw is mat, and temperature 3, where low-probability tokens such as spaceship start appearing. With top-K 3, only three candidates remain. Top-P 0.9 removes the long tail while preserving the relative probabilities of the remaining tokens.

§2.2Top-K and top-P: what each cutoff counts

Both filters work on the same list, sorted most likely first, and both throw away everything below a line. They differ in what the line is measured in. Top-K counts rows. It keeps the first K of them and does not look at the numbers. Top-P adds up probability. It walks down the list adding each row's probability to a running total, and it stops at the first row that brings the total to P or above. That row stays in. Every row after it is cut. So top-P 0.7 does not mean "tokens with more than 70%". It means "enough of the top tokens to cover 70% between them".

The number of survivors under top-P is not a setting. It falls out of how the probability is spread at that step, which is another way of saying how confident the model is. When one token holds most of the probability, the running total crosses P almost at once and one or two tokens survive. When the probability is spread thin, the walk has to go a long way before the total reaches P, and dozens of tokens survive. Pick a context below and watch the running total column.

Same K and P, three different steps. Top-K returns the same count every time. Top-P moves with the shape of the distribution. Click a row to load it above. I made up these three distributions to show the cases. A real model produces numbers like these at different steps of the same sentence. Some steps have one obvious continuation and others have many.

§2.5The last step: rolling a weighted die

Temperature, top-K and top-P only decide which tokens are still in the running. None of them picks the token. The pick is a weighted random draw over whatever survived. Lay the survivors end to end along a line from 0 to 1, each owning a slice as wide as its probability, then draw a random number and see where it lands. A token holding 47% of the line gets hit 47% of the time. torch.multinomial performs exactly this draw, and do_sample=True is what switches it on.

There is no slider for this step. do_sample is the only control, and it is binary. With False, generation takes the argmax and ignores temperature, top-K and top-P entirely. With True, you get this draw. Argmax is still reachable from inside sampling — top-K 1 leaves one survivor owning the whole line, and temperature near 0 stretches the leader across almost all of it. For repeatable output without giving up sampling, set the random seed instead. set_seed(42) makes the same prompt and settings produce the same rolls every run.

§3Generation playground

This browser-based bigram model is counted from a small corpus about cats, dogs and mice. It predicts the next word from only the current word. Its autoregressive loop, EOS stopping and sampling steps also appear in GPT-2, which uses a much larger vocabulary and context.

§4The same knobs in code

Hugging Face exposes these controls through model.generate(), as used in the Colab exercise.

from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed

tok   = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

ids = tok("The cat sat on the", return_tensors="pt").input_ids

set_seed(42)  # same seed + same settings = same rolls, every run (§2.5)

out = model.generate(
    ids,
    max_new_tokens=40,        # cap on the autoregressive loop (§1)
    do_sample=True,           # False → greedy argmax every step
    temperature=0.8,          # <1 sharper, >1 flatter (§2 slider 1)
    top_k=50,                 # keep the 50 most likely tokens (slider 2)
    top_p=0.95,               # nucleus: smallest set covering 95% (slider 3)
    eos_token_id=tok.eos_token_id,  # the stop token
)
print(tok.decode(out[0]))

generate() hides the actual sampling work. Written out by hand, one sampling step is a handful of tensor calls. The method names below keep coming back in later chapters, so they are worth recognizing now.

# one hand-rolled sampling step (top-K variant)
logits = model(ids).logits[0, -1]                # a score for every token (§1, step 2)
probs  = torch.softmax(logits / 0.8, dim=-1)    # temperature, then scores → probabilities
vals, idx = torch.topk(probs, k=50)             # keep the 50 most likely (top-K)
vals   = vals / vals.sum()                      # renormalize the survivors
pick   = torch.multinomial(vals, num_samples=1) # the weighted draw from §2.5
next_id = idx[pick]

# for top-P, replace the topk lines with:
#   sort descending, cum = torch.cumsum(sorted_probs, dim=-1),
#   and cut where cum passes P

§5Test yourself

Questions on sampling, generation and the weighted draw. Each answer includes an explanation.

§6Key takeaways

§1Looking inside with print(model)

Load any open model in PyTorch and print it. What comes back is the module tree, every stored component nested the way data flows through them. Click any module to see what it does, how many of the 124M parameters it holds, and where it fits. The block that repeats 12 times is why GPT-2 small is called a 12-layer model.

§2Where the 124 million parameters live

A parameter is a stored number learned during training. GPT-2 small is quoted as both 124M and 163M parameters. The toggle below explains the gap: the output head reuses (ties) the token-embedding matrix, so those 38.6M numbers get counted once or twice depending on convention.

The layernorms account for only 0.03% of the parameters even though they are essential to training. Almost all the stored capacity sits in the MLPs and the embeddings.

§3Compare with TinyLlama

print() works on anything from Hugging Face. This is TinyLlama, a 1.1B Llama-style model from 2023, printed next to GPT-2. Click a row. The skeleton is the same, embeddings and then a repeated block with attention and an MLP inside. Each difference is a named upgrade that came out of four more years of experiments.

§4The component list as a syllabus

Every component in the tree maps to a chapter of this lab. The training machinery (loss functions, backpropagation, initialization) is missing from the tree because it is not stored in the model; you only need it while building one. Each section also compresses to roughly one line of code, shown under its title below. Most of these lines mean nothing yet. They start making sense chapter by chapter, which makes this list a decent progress check. Green marks what this lab covers so far.

§5Why these components? Ablation.

There is often no first-principles answer for why GPT-2 uses layernorm, GELU or another particular component. The field runs ablation tests instead: remove a component, retrain, compare. If the model gets worse, the component stays. GPT-2's component list is a record of which options performed best in those comparisons around 2019.

§6Test yourself

Ten questions on reverse-engineering. Instant feedback, explanations included.

§7Key takeaways

§1A machine with three knobs

f(x) = w·x + b. The input x arrives from outside, or from another neuron. The weight w and bias b are the two numbers the perceptron stores. Training adjusts millions of these stored values. The unit dates to Rosenblatt, 1958. The board below draws the signal as it travels: the input enters on the left, the wire scales it by the weight, the bias joins from above, and the needle on the right shows the result.

§2One perceptron, every input at once

Sweep the input from −5 to 5 and plot the output. A perceptron is a straight line. A line has two properties, slope and intercept, and the perceptron's two stored numbers map onto them directly: w is the slope and b is where the line crosses the y-axis. The presets cover the cases worth seeing at least once.

§3The landscape: finding an edge

Now vary two things at once: input across, bias down, weight on the slider. Color every output and a pattern shows up — negative outputs on one side, positive on the other, a straight boundary between them at zero. This is a perceptron working as a classifier. Notice the two limits, though. The boundary is always straight. And the zero cutoff is a convention we chose; nothing in the unit enforces it.

§4The same thing in code

The same unit in Python. The class stores w and b; activate computes the output. PyTorch has this built in (nn.Linear is rows of these run in parallel), but writing one by hand once makes the stored-numbers idea concrete.

class Perceptron:
    def __init__(self, weight, bias):
        self.weight = weight    # stored — the "memory"
        self.bias   = bias

    def activate(self, x):
        return self.weight * x + self.bias   # f(x) = w·x + b

p = Perceptron(weight=1, bias=3)
for x in range(-5, 6):
    print(x, p.activate(x))   # -5 → -2,  -4 → -1,  …  5 → 8

§5Test yourself

Ten questions on perceptrons.

§6Key takeaways

§1The impossible fit

To see why nonlinearity is needed at all, try fitting the amber target, |x|. Absolute value shows up even in language: "great" and "awful" sit at opposite ends of sentiment, but both score high on intensity, and intensity is an absolute value. First try to lay a single perceptron's line on the target. Then switch to two ReLU-wrapped perceptrons added together.

§2Why the rule has to change

A neural network is only ever copying a relationship that already exists in the data. Input in, output out, and training pushes the weights until the model's outputs match the real ones. So the model needs a place where its rule changes only if the thing it is copying has one. That makes this a question about the data before it is a question about the network.

A line commits to one rate. Every extra unit of input adds the same amount to the output, whether the input is 1 or 1,000, negative or positive. Very few real relationships work like that. Heating cost against outdoor temperature reverses direction once the day is warm enough to need cooling. A drug dose does nothing below some amount, helps above it, and harms past a higher one. Income tax charges nothing below a threshold, then a rate, then a higher rate. Each of those has a point where the rule changes, and a line has no such point.

Income tax is the cleanest of the three, because a progressive tax is a sum of ReLU units. The brackets here are made up for the example: nothing on the first 10,000, then 20%, then a further rate above a second threshold. Drag the income and compare the bracketed tax with a flat tax, which is a single perceptron. Then try to find a flat rate that matches the brackets at every income.

Each bracket is one ReLU unit. Written in chapter 3's form, relu(w·x + b), the weight is the extra rate that starts at that bracket, and the bias is minus the rate times the threshold, so the unit crosses zero exactly at the threshold. The government did not use a flat line because it wanted low earners to pay a smaller share. That wish is the same thing as "the rule changes at 10,000". A model that can only draw lines puts one line through the middle and is wrong for everyone. The corners are in the data, so the model has to be able to draw corners.

For language, the rule changes with every word. This is the reason that matters for a language model. Take a scorer that gives each word a number and adds the numbers up, which is a perceptron over words. "good" scores +2 and "bad" scores −2. Now pick a score for "not" and try to get all four sentences right.

Whatever number "not" gets, it moves both sentences the same distance in the same direction. The real rule is that "not" flips the sign of the word after it. Flipping is not shifting, and a line can only shift. The two ReLU units get all four right because each unit stays switched off until "not" is present, and then it subtracts twice the word's score, which is what a flip is. The bias of −4 is what keeps the unit off. It is the same job the tax threshold does.

Put more generally, in a line the effect of each input is fixed and independent of every other input. The score for "good" cannot depend on whether "not" is nearby. But language is made of words changing what other words mean. "Bank" next to "river" and "bank" next to "loan". "Great" in "a great movie" and in "a great disappointment". Each of those is the rule for one word changing because of the words around it, and a model built only from lines has no way for one input to change the effect of another. The corner is the smallest piece that can do it. On one side of the corner an input counts, and on the other side it does not. Everything built in later chapters still relies on these corners to let context change meaning.

§3The perceptron with ReLU

Chapter 3's knob machine, with one part bolted on after the perceptron: ReLU(x) = max(0, x). Positive signal passes straight through; negative signal is blocked at zero. Drive the perceptron negative and watch the gate close.

§4The curve gallery

Every activation function is a shape you pass the perceptron's output through. The x-axis runs −4 to 4 because that is where these functions actually operate: perceptron outputs tend to be standard-deviation-sized, and 99.7% of a normal distribution falls within ±3. Select a function to spotlight it and read what its shape does; the rest stay as gray reference curves. Drag the probe for exact values.

§5Landscapes, revisited

This repeats the chapter 3 landscape (input across, bias down), with the output passed through an activation. Under ReLU the whole negative region becomes one flat plane of zeros. Leaky ReLU keeps a faint slope there. GELU rounds the transition. The flat regions matter later: learning works by following slopes, and a perfectly flat region has no slope left to follow.

§6Which one to use

The field's answer changed three times in eight years, and there is no full account of why each approach performed best. The short history:

There is no correct activation function and no accepted theory that picks one. The working method is to run the ablation and keep the better number. For a small GPT-2-style model, ReLU² is a reasonable default for exactly that reason. At 124M parameters on a modern GPU the choice barely moves the result anyway; the differences show up at scale.

§7Test yourself

Thirteen questions on activation functions.

§8Key takeaways

§1Eight cores against fifteen thousand

A desktop CPU has about 4 physical cores, 8 with hyper-threading. An H100 GPU has about 15,000 cores built for exactly this kind of math. For a handful of operations the CPU is fine. At 32 million elements, GELU took 0.2 milliseconds on GPU and 12–20 seconds on CPU in the benchmark. Move the element-count slider to see when the difference starts to matter.

§2The ping-pong problem, and fused kernels

GELU contains nine math operations: a cube, additions, tanh and multiplications. In a naive implementation, each operation runs as a separate GPU kernel and pays its own launch and data-transfer overhead. Those overheads cost more than the math. A fused kernel combines all nine operations into one GPU program. Toggle the two implementations to compare where the time goes.

§3Six ways to write the same line

These implementations all calculate the same activation function. Click a level for its trade-offs. The bars use the benchmark results. The differences are small at 1K elements and become large from 32K elements onwards.

The workshop model uses torch.compile to keep the code readable. Measured against the fastest options above, that choice roughly doubles the GPU bill. At frontier scale, teams write lower-level GPU code because another 5% of performance is worth the engineering time.

§4Test yourself

Ten questions on GPU coding.

§5Key takeaways

§1Several inputs, one neuron

With more than one input the formula barely changes. Each input gets its own weight, the products are summed, and one bias is added at the end: output = w₁x₁ + w₂x₂ + b. A negative weight makes an input push the output down instead of up.

§2Two layers: the signal cascades

Now two neurons, A and B, each read both inputs, and a third neuron reads A and B. Every neuron has its own weight for every input, so two inputs into two neurons already means four weights, and the output neuron adds two more. Each hidden neuron passes through a ReLU gate before its value travels on. Biases are fixed at zero here to keep the board readable. Drag any slider and follow the change through both layers.

Two things are worth noticing. A closed gate is a dead end: whatever weight the output neuron has for that hidden neuron, it receives zero from it. And the output neuron never sees the inputs directly. It only sees what A and B chose to pass on, which is why the second layer can compute things the first layer cannot.

§3XOR: the job one layer cannot do

XOR means "either, but not both". No single perceptron or single layer of perceptrons can compute it. This version uses two hidden neurons feeding a third. The first hidden neuron, "either", fires when at least one input is on. The second, "both", has a bias of −1, so it only fires when both inputs are on. The output neuron adds "either" with weight 2 and "both" with weight −6, so the moment both inputs are on, the second neuron cancels everything. Flip the switches and follow the signal.

Every extra layer lets the network compute functions that the previous layer could not.

§4Matmul, and the grow-then-shrink pattern

Matrix multiplication calculates all of those weighted sums at once. It is written as @ in PyTorch and MMULT in a spreadsheet. The dimension rule is (a×n)·(n×b) = a×b, so the inner dimensions must match. Multiply a 4-wide input by a 4×16 matrix and you get 16 activations. Multiply those by 16×4 and you are back to 4. An LLM's MLP widens the vector, applies the activation and narrows it again. Move the expansion factor to see how this changes GPT-2's parameter count.

Geva (2021) describes the wide middle as a scratchpad where the model can represent more patterns before narrowing the vector again. These large matrices also store much of the model's knowledge. MLPs account for 45% of GPT-2's parameters.

§5The real thing, in PyTorch

This is the MLP used in the workshop model. It has two linear layers with ReLU² between them. It has no bias terms, which is common in modern LLMs.

class GPT2StyleMLP(nn.Module):
    def __init__(self, dim=768, expand=4):
        super().__init__()
        self.up   = nn.Linear(dim, dim * expand, bias=False)  # 768 → 3072
        self.down = nn.Linear(dim * expand, dim, bias=False)  # 3072 → 768

    def forward(self, x):
        return self.down(F.relu(self.up(x)).square())  # grow, bend (ReLU²), shrink

# ~2.36M parameters per matrix, ~4.7M per MLP — ×12 blocks in the model

§6Test yourself

Ten questions on MLPs.

§7Key takeaways

§1Score one preparation-time prediction

Use the same made-up restaurant example as chapter 8. The model estimates preparation time as minutes per item × item count + starting minutes. Those two settings are the weight and bias from chapter 3. At 2 minutes per item and 3 starting minutes, a four-item order gets a prediction of 2 × 4 + 3 = 11 minutes.

The recorded time is 15 minutes. This is the target used by the training code to check the prediction. It is not an extra input to the model. The prediction is four minutes too low.

Chapter 8 uses half the squared error, so this example's loss is 16 ÷ 2 = 8. Dividing by two makes the later derivative simpler. It does not change which prediction has the lowest loss. This score is not eight minutes; the original miss was four minutes.

prediction = model(itemCount)       // 11 minutes
error = prediction - recordedTime   // 11 - 15 = -4
loss = error * error / 2            // 8

Calculating the loss has not changed either stored setting. It has only scored the prediction. Chapter 8 adds the backward calculation and the update.

§2Combine errors without letting them cancel

Suppose two orders receive these predictions. The numbers are teaching examples, and the slider edits the second prediction directly to isolate the scoring calculation. At the starting values, one prediction is four minutes too low and the other is four minutes too high. Their signed errors average to zero even though both answers are wrong.

Mean squared error (MSE) squares each error before averaging. At the starting values, that is (16 + 16) ÷ 2 = 16. Zero now requires every prediction to be right. Opposite signs cannot cancel after squaring.

Root mean squared error (RMSE) takes the square root of that average. The square root of 16 is 4 because 4 × 4 = 16. RMSE puts the score back into minutes, making its size easier to read. It still gives larger misses extra influence; it is not simply the average number of minutes missed.

Fixing the second prediction lowers MSE from 16 to 8, even though the average signed error moves away from zero. That is the behaviour we want from an error score. The first order is still wrong, so the loss remains positive.

§3Compare different model settings

Now calculate predictions from the model again. Keep two recorded orders fixed: four items took 15 minutes, and one item took 4 minutes. Each cell below tries a different pair of stored settings, makes both predictions, and scores their RMSE.

Columns change minutes per item. Rows change starting minutes. Paler cells have lower loss. Select a cell to see its predictions and errors. The outlined starting cell uses the same 2 and 3 as chapter 8.

This map is a loss landscape. If we plotted each cell's loss as a height, the lower-loss settings would sit lower down. Each cell uses different model settings with the same recorded answers. Training tries to find settings with lower loss without testing every possible cell.

§4Score a choice among possible answers

Preparation time is a numerical prediction. Suppose instead a model reads an order note and chooses the dish: soup, pizza or salad. These are categories. Assigning them IDs 0, 1 and 2 does not make pizza halfway between soup and salad, so subtracting those IDs would be a poor error measure.

The model gives a probability to each dish. In this made-up example it assigns soup 40%, pizza 35% and salad 25%. The recorded order says soup. Soup has the highest probability, so the model's top choice is right, but it gives 60% of the total probability to other answers.

A right/wrong score cannot distinguish that prediction from one assigning soup 90%. A probability-based loss can. Cross-entropy gives a smaller penalty when the model assigns more probability to the recorded answer. The next section explains its calculation.

Try 40%, then 90%. Both can produce the right top choice, but the second gets a smaller loss. Then try 1%. The model gave almost no probability to the recorded answer, so the penalty is large.

Keep the recorded answer at 40% and move the remaining-probability slider. The top choice may change, but this example's loss stays the same. That loss uses the probability of its recorded answer. Probabilities still have to total 100%, so increasing the recorded answer's share requires reducing the others' combined share.

§5Unpack “one-hot target” and “−log”

One-hot is a way to write down the answer

Keep the dish order fixed as [soup, pizza, salad]. To record soup, write [1, 0, 0]. The 1 marks soup as the answer for this example. The zeros mark the other dishes. To record pizza, write [0, 1, 0]. This list is called a one-hot target because exactly one position is 1.

This is an answer marker supplied by the data. It is separate from the model's probabilities, such as [0.40, 0.35, 0.25]. A target of 1 for soup does not mean the model already assigned it 100%.

−log converts a probability into a penalty

Read −log(p) as “take the natural logarithm of probability p, then change its sign.” A logarithm is a mathematical function, available as Math.log in JavaScript. Use 0.40 for 40%, not the number 40. You do not need to calculate logarithms by hand to use this loss.

Between zero and one, the logarithm is negative. Changing its sign gives a positive penalty. At probability 1 the penalty is zero. As the probability approaches zero, the penalty grows without a fixed upper limit. This strongly penalizes giving almost no probability to an answer that actually occurred.

The curve also makes relative changes comparable. Raising the correct answer's probability from 10% to 20% reduces loss by about 0.69. Raising it from 40% to 80% reduces loss by the same amount. Both changes doubled that probability. This is different from simply subtracting the probability from 1.

The target marker selects which penalty to count

Calculate a −log penalty for each probability. Multiply each penalty by its answer marker, then add the results. Multiplication by zero gives zero. Multiplication by one keeps the number. Try changing the recorded dish below and watch the 1 move to a different row. These controls also update the example in §4.

That is what “only one term survives” means. For a soup record, the pizza and salad rows contribute zero to this sum. The soup row contributes −log(probability of soup). A pizza record would select the pizza row instead. For a batch, calculate one loss per recorded example and average those losses.

§6Use the loss in the training loop

A language model makes a similar choice over tokens. Given a prefix such as “Please bring the”, the next token in a recorded text supplies the training answer. Many continuations may be reasonable, but this example scores the probability of the token that actually followed. Training repeats this across many pieces of text.

The next-token loss rewards assigning more probability to observed continuations. It does not directly test truth, usefulness or every valid way to finish a sentence. Lower training loss also does not establish better performance on new text; that needs a separate check.

scores = model(inputs)              # one raw score per possible answer
loss = F.cross_entropy(scores, targets)
loss.backward()                    # calculate gradients
optimizer.step()                   # update stored settings

§7Check your understanding

Ten questions about the scores and what they mean. No logarithm calculations are required.

§8Keep these distinctions clear

PlaygroundAssemble a model that can learn

A neuron can calculate a prediction before it can learn anything. Adding more neurons makes a larger prediction function. To learn from mistakes, the surrounding program also needs a recorded answer, a loss calculation, a backward calculation and an update rule.

Use a made-up four-item order that took 15 minutes. Add one component at a time below. The blue circuit is the prediction function. The boxes underneath show the training work around it. They preview the available calculations; the trace button steps through them. This small example runs independently of the widgets later in the chapter.

Try removing the optimizer after assembling the loop. Select “Backpropagation” above and trace the available steps. Gradients can be calculated, but the stored settings stay unchanged. Add the optimizer again to make saving changes possible. During ordinary use, the trained prediction function works without a target, loss or backward pass.

§1The model is still the function you already know

Suppose we want to estimate restaurant preparation time from the number of items in an order. This is a made-up teaching example, not a measurement from a real kitchen. Start with a very simple function:

minutesPerItem = 2     // a stored number we can change
startingMinutes = 3    // another stored number we can change

predict(itemCount):
    return minutesPerItem * itemCount + startingMinutes

This is the perceptron from chapter 3. “Minutes per item” is its weight. “Starting minutes” is its bias. For four items it returns 2 × 4 + 3 = 11 minutes. The multiplication and addition stay the same during training. We change the two stored numbers.

In chapter 4 you added activation functions, and in chapter 6 you connected several neurons. Those larger models have more stored weights and biases, but they still calculate an output from an input. Chapter 5's GPU operations make those calculations faster. Training adds a way to adjust the stored numbers using examples.

Learning means keeping those adjusted numbers for later predictions. The program still runs the same prediction code. During ordinary use, there is no recorded answer yet, so making a prediction does not by itself perform another training update.

§2Find out what changing one number would do

We need the prediction to rise from 11 toward 15. Raising either setting would help for this order, but they have different effects. Add 0.1 to minutes per item and the prediction rises by 0.4, because there are four items. Add 0.1 to starting minutes and the prediction rises by only 0.1.

Try a small change to one setting while keeping the other fixed. The three rows below are separate tests of the same starting model. No change is saved.

Try the nine-minute record. The model now predicts too much. The useful direction reverses. A weight is not inherently too high or too low; its useful direction depends on the example, the target and the model's current settings.

The number that describes that effect is a gradient

The test above asked whether a small increase makes the loss rise or fall. A derivative describes how fast that loss changes at the current setting. For a model with many settings, we collect one such number for each weight and bias. That collection is the gradient. You will also hear “the gradient of this weight” used for its individual number.

For example, increase minutes per item from 2 to 2.01. The prediction becomes 11.04, so the miss is 3.96 minutes. Squaring that miss and halving it gives a loss of 7.8408, down from 8. The setting rose by 0.01 and loss fell by 0.1592. Dividing the loss change by the setting change gives about −15.92. That describes how strongly this setting affects loss near its starting value.

Make the test change smaller and that rate approaches −16, the weight's gradient at the starting value. The same calculation for the bias gives −4. A negative sign means a small increase in the setting would lower the loss. The weight has four times the effect here because it is multiplied by four items. A loss of 8 is a score, not eight minutes. And a gradient of −16 is a rate of change of that score, not an instruction to add 16 to the weight.

Backpropagation computes these numbers without trying every setting separately

For two settings, testing small changes is easy. For millions of settings, running the whole model again for every test would be expensive. Instead, each operation in the calculation supplies its own rate of change. Multiplying a number by four multiplies a small change in it by four. Adding a bias passes a small change straight through.

Backpropagation starts at the loss and follows these effects backward through the calculation. It works out a gradient for each stored number. The name comes from this backward calculation. The model's prediction still runs forward from input to output.

§3Use the gradients to make a small update

We have calculated how each setting affects loss. We still need a rule for actually changing it. The simplest rule is gradient descent: subtract a small multiple of the gradient from each setting. The multiplier is called the learning rate. We choose it in the training setup.

Use a learning rate of 0.02. Multiply the weight's gradient, −16, by 0.02 to get −0.32. Subtracting −0.32 adds 0.32 to the weight. The bias's gradient of −4 gives a change of +0.08 in the same way. So minutes per item becomes 2.32 and starting minutes becomes 3.08. The new prediction is 2.32 × 4 + 3.08 = 12.36 minutes. It is closer to the recorded 15.

Pause after “Work backward”. The stored settings have not changed. Backpropagation has only calculated the gradients. Press the next button to apply the update. The component that applies an update rule is called the optimizer; this example uses the simple gradient-descent rule.

After an update, the prediction and its error can change. Calculate fresh gradients at the new settings before making another correction. The training loop repeats this process for every update.

Why take small steps when we already know the answer?

For this single order, we could set the starting time to 7 and keep minutes per item at 2. That would give exactly 15. But it tells us nothing about whether those settings work for other orders. In a larger network, each setting also affects the answer through other settings and activation functions.

A gradient describes what helps near the current settings. A large change can go past the useful region. At four items, reset and try 0.005 for slow progress, then 0.12 for a step that overshoots and makes the error grow. Keep watching the loss across several updates. Lower is better; a bouncing prediction does not necessarily mean the error is shrinking.

§4An earlier neuron affects the answer through later ones

In chapter 6, you connected neurons. An early weight no longer changes the final answer directly. It changes a hidden neuron's output, which is used by a later neuron. Backpropagation has to account for both effects.

Use a small two-branch network so we can see the whole calculation. The numbers below are unitless teaching values, separate from the restaurant example. Input 1 reaches neurons A and B. A outputs 1, B outputs 0.5. The final neuron adds A's output, subtracts half of B's output, and adds 0.1. Its prediction is 0.85; the target is 1.

At these settings, increasing A's incoming weight by 0.1 would raise A's output by 0.1. A's outgoing weight is 1, so the final prediction would also rise by 0.1, to 0.95. That is closer to the target. To reach the earlier weight, the backward calculation follows the same operations in reverse order.

Multiplication can strengthen or reverse an effect

Change A's influence to 2. A small change in A's output now has twice the effect on the final answer. Change it to −1 and raising A's output lowers the final answer. So “the prediction is too low” is not enough to decide that every earlier weight should increase. The backward calculation needs the actual connections.

A closed ReLU blocks a small change

From chapter 4, ReLU returns zero for a negative input. Set A's bias to −2. Its value before ReLU is now −1. A small increase would still leave it negative, so its output would still be zero. That small change would have no effect on the final answer. The loss therefore gives A's incoming weight no gradient for this example.

The other branch and the final bias can still receive updates. Another input could also make A active. A zero gradient here describes this particular calculation; it does not mean that neuron can never be useful.

The mathematical name for combining the effects along a path is the chain rule. You multiply the rates through successive operations. If an earlier value affects the answer through two branches, both effects count, so their returning contributions are added. The same bookkeeping extends to many layers.

§5Different examples can ask for different changes

Return to the restaurant model at 2 minutes per item plus 3 starting minutes. The four-item order took 15 minutes, so its prediction of 11 is too low. Add a second made-up record: a one-item order took 4 minutes. Its prediction of 5 is too high.

Both orders use the same stored settings. The first example favours increasing them; the second favours decreasing them. A batch is a group of examples considered before one update. Compute their gradients at the same current settings, then average them. Each update can be a compromise that helps the average loss even if it makes one individual prediction worse.

A training program goes through the recorded examples in batches and updates the settings after each batch. One complete pass through all the training examples is an epoch. For 7,000 examples in groups of 64, that is 110 updates per epoch, including the last smaller group.

Small batches can disagree more from one update to the next. Larger batches average over more examples, but use more memory and produce fewer updates per epoch. Chapter 5's GPU work matters here because many examples can be calculated together. A larger batch is not automatically a better way to learn.

§6The optimizer chooses how to use the gradients

The training code now has a list of numbers: the loss gradient for every weight and bias. An optimizer reads that list and changes the settings. Different optimizers use different rules. Backpropagation supplies the gradients in all these cases.

To compare their behaviour without another network, use a model whose output is just one adjustable number. It starts at 2 and its target is 0. Press update several times. Watch how far each rule moves the number on each click, especially once momentum has a history to use.

Being faster on this simple problem does not establish which optimizer is best for a larger model. The useful distinction is what information each rule uses to choose an update. Other names such as Adagrad, RMSProp and Muon describe other update rules; none replaces the need to calculate gradients.

The learning rate can change during a run

Instead of keeping the multiplier fixed, a training program can start small, increase it for a while, then gradually reduce it. The initial increase is called warmup. It can help keep early updates controlled while the model and the optimizer's running records settle. Later, reducing the rate makes smaller adjustments possible.

Move through the planned updates below. This schedule depends only on the update number. It does not look at the target or decide that the model now understands the task. The rising part lasts 10 updates, followed by a smooth decrease to one tenth of the peak rate by update 100.

§7Put the whole process together

The restaurant example let us inspect two settings. Now train a network with thousands of them. Its task is to add two whole numbers and predict the remainder after dividing the sum by five. For 7 and 9, the sum is 16: three groups of five, with 1 left over. The possible answers are 0, 1, 2, 3 and 4.

We can calculate the answer with ordinary code. That makes it convenient to generate examples whose answers we know. The learning experiment is to give the network the two inputs, compare its prediction with our calculated answer, and let the training loop adjust its weights. The network itself is never given the remainder formula.

Start with “Train one epoch”. That processes each of the 7,000 training examples once. Watch the two accuracy figures below. Accuracy is the percentage of answers that were right. Around 20% is what we expect from guessing among five possibilities. Then train 10 more epochs and inspect a grid cell.

Read the result before changing the settings

If training accuracy rises while validation stays near 20%, the current settings are fitting the seen pairs without helping much on the unused ones. When validation also improves, there is evidence that the learning transfers to those pairs. Neither result establishes that the model could handle an entirely different range of inputs.

Try reducing the learning rate from 0.01 to 0.001, then run the same number of epochs. Compare how far it gets. Next, change examples per update from 64 to 256. There are now fewer updates per epoch, so check the update count as well. Every curve here comes from actual browser computation. Some settings can stall or make the results worse.

§8The training program runs the same sequence

The model function calculates predictions. The surrounding training code supplies examples, checks answers and calls the optimizer. That surrounding loop is what causes learning to happen.

for each group of training examples:
    clear the previous gradients
    predictions = model(inputs)
    loss = compare(predictions, recordedAnswers)
    calculate how each weight and bias affects loss
    update the weights and biases using those gradients

In PyTorch, loss.backward() does the backward calculation and optimizer.step() applies the changes. PyTorch can work backward because it records the operations used to make the predictions and keeps intermediate values it will need. This feature is called autograd. You do not have to write all the derivatives yourself.

The same arrangement applies to next-token training. The input is some text, the recorded answer is the next token in that text, and the model assigns probabilities to possible tokens. The loss checks the probability of the recorded answer. Repeating updates changes the stored weights so future predictions fit the training examples better. Checking separate text tells us whether this improvement transfers.

Use Colab after the arithmetic feels clear

The browser exercise is enough to experiment with this learning loop. Running the chapter 8 Colab exercise afterward is useful for seeing the same steps in real PyTorch code. It adds practice with the library; it is not a prerequisite for understanding the process.

For a useful first code experiment, print one weight before backward(), after backward(), and after step(). The first two values should match. The last can change. Then inspect that weight's .grad value between backward() and step(): that is the gradient awaiting an update. See the PyTorch explanation of autograd for the library details.

optimizer.zero_grad(set_to_none=True)  # clear the previous gradients
logits = model(inputs)                 # calculate scores for possible answers
loss = F.cross_entropy(logits, targets) # compare with the recorded answers
loss.backward()                        # calculate gradients
optimizer.step()                       # change the stored weights and biases

§9Check your understanding

Twelve questions about what happens and why. The main explanation contains everything needed; no derivative calculations are required.

§10What changes when the model learns

§1Five formats, five jobs

Click a format for what it is for. The one to be careful with is .pth: loading one executes a serialization machine that can run arbitrary code, so it is fine for files you saved yourself and dangerous for files from the internet.

§2Checkpointing arithmetic

A 20-hour pre-training run can die at hour 19, so you save the model as you go. The checkpoint file is about 1.4GB and is saved every 100 steps across a 20,000-step run. Keeping every checkpoint uses about 280GB. Keeping only the latest uses about 1.4GB.

§3The code

state_dict() holds only the learned parameters. If training will continue later, save the optimizer state too — chapter 8's per-parameter statistics are part of where the run left off.

# save (weights only — training is done)
torch.save(model.state_dict(), "workshop-v1-pretraining.pth")

# load: build the architecture, then fill in the learned numbers
model = GPT2StyleModel()
model.load_state_dict(torch.load("workshop-v1-pretraining.pth"))

# or pull a published model straight from Hugging Face
model = AutoModelForCausalLM.from_pretrained("JustinAngel/workshop-v1-pretraining")
out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=50)
# "The meaning of life is to live as a human being…"

§4Test yourself

Eight questions on saving and loading.

§5Key takeaways

§1Two ways to draw a random number

A uniform draw gives every value in a range the same chance. A normal draw follows the bell curve: values cluster around the mean, and the standard deviation sets how tightly values cluster. About 68% land within one SD, 95% within two, and 99.7% within three. Initialization formulas choose a standard deviation that keeps values stable through repeated matrix multiplications.

§2Ten rounds of matmul and ReLU

This experiment starts with a random 5×5 matrix, multiplies it by a fresh random matrix, apply ReLU, and repeat ten times, tracking the average value each round. Pick an initialization and rerun it a few times. Uniform explodes. Plain normal collapses or explodes depending on the roll. He initialization — standard deviation sqrt(2 / fan-in), built to compensate for ReLU zeroing half the values — holds steady.

§3Initialization and convergence

The reported experiment trains the same deep network (220 → 512 → 512 → 256 → 256 → 128 → 19, on A×B mod C) four times, changing only the initialization and activation. Epochs to full convergence:

The formulas are small: He is sqrt(2 / fan_in) for ReLU networks, Xavier is sqrt(2 / (fan_in + fan_out)) for tanh and sigmoid. In PyTorch they are one call each from torch.nn.init. Karpathy's line fits here: baby zebras don't randomly spasm their muscles — good initialization is wisdom wired in before the first step of learning.

§4Test yourself

Ten questions on initialization.

§5Key takeaways

§1Signal survival through depth

Each layer multiplies the signal by something a little off from 1. Without residuals those factors compound. By layer 30 the signal may have drifted to a millionth or a million times its size. With residuals the input is added back after every block, so the stream stays near its original scale. Slide the depth and compare.

In the reported A×B mod C experiment at 32 layers, accuracy increased from 33% to 57% in 50 epochs without residuals, with loss spikes on the way. With residuals, it reached 100%. At 72 layers without residuals, accuracy never moves at all — inner activations explode to around 10⁸ and nothing can be learned.

§2The block, drawn out

This residual MLP block has the same structure used in a transformer layer. The signal takes two paths: through the block's matmuls and activation, and straight past it on the shortcut. They meet in an addition.

Variants exist. Scaled residuals multiply f(x) by a constant like 0.5 so magnitudes don't double at every add. Gated (highway) residuals learn how much to let through. Concatenation residuals keep input and output side by side, which attention uses internally. And the Kimi team's recent attention residuals let a layer reach back to any earlier layer, not just the previous one. The plain addition remains the common implementation.

§3Why an addition fixes training

Research offers two useful explanations.

§4Test yourself

Ten questions on residuals.

§5Key takeaways

§1RMSNorm calculation

RMSNorm divides each value in a row by the root of the mean of their squares (plus a tiny ε so the division can never hit zero): x / sqrt(mean(x²) + ε), times a learned scale. RMSNorm does not center the row or calculate its variance. Move one value to see how the shared divisor rescales every value in the row.

§2LayerNorm, BatchNorm, RMSNorm

Three norms, two operations between them. Scaling pulls values toward a fixed bound. Centering shifts the mean back to zero. Click a row for who does what and why LLMs settled where they did.

§3Pre-norm and post-norm

The same norm can sit before the block's work (pre-norm) or after the residual addition (post-norm). The 2017 transformer used post-norm; Xiong (2020) showed it trains unstably, and the field moved. The intuition: normalizing right after the residual add throws away part of the residual you just fought to preserve. Toggle the placement and read the reported results.

At 72 layers, every pre-norm variant converged and every post-norm variant collapsed in the experiment. In the post-norm runs, the first ~60 layers were normalized into learning nothing while the last few strain to compensate. The activation charts show pre-norm holding values in the 0–1 band for the entire run.

§4Test yourself

Ten questions on normalization.

§5Key takeaways

§1Watching overfitting, and fixing it

Overfitting appears as a widening gap between training and test accuracy. The model is learning noise that the test set does not share. Step through the weight-decay sweep to see the gap close. Weight decay subtracts a small fraction from every weight at every update, so the network can only keep weights it keeps re-earning.

There is also a floor under all of this. The Chinchilla paper measured an irreducible loss of about 1.69 across 400 training runs — no amount of compute, parameters, or data pushed language loss below it. With 10% noise injected, this dataset has its own floor: 90% test accuracy is the honest maximum, and the sweep's winner reaches it.

§2Dropout: training with pieces missing

Dropout silences a random set of neurons at every training step — a different set each step, so no single weight can become indispensable. The survivors get scaled up by 1/(1−p) to keep the magnitudes right, and at inference nothing is dropped at all. Regenerate the mask a few times.

GPT-2 trained with 10% dropout in 2019. Modern LLMs skip it entirely: dropout counters the overfitting that comes from showing data many times, and LLM pre-training sees each token roughly once. In this sweep, dropout helped a little but performed worse than weight decay. The workshop model keeps a 10% dropout rate.

§3Gradient clipping: capping the step, keeping the direction

Occasionally a batch produces huge gradients that would wreck the weights in one step. Clipping caps them, and how you cap matters. Value clipping clamps each gradient separately, which changes their proportions. Norm clipping scales the whole set by one shared factor, so the update shrinks but still points the same way. Compare the two methods on three gradients.

The proportions determine the update direction in a high-dimensional parameter space. Value clipping changes that direction, so norm clipping is used instead. In the reported runs, adding norm clipping on top of weight decay reached the 90% target about 25% fewer epochs in. In PyTorch it is one line between loss.backward() and optimizer.step(): nn.utils.clip_grad_norm_.

§4Test yourself

Ten questions on regularization.

§5Key takeaways

§1What the exponent does

e^x maps every negative number between 0 and 1 and increases quickly for positive numbers. Negative logits need no special handling, while gaps between large logits become more pronounced. Move the probe to inspect the curve.

§2Softmax against the naive alternative

A linear alternative shifts every value above zero and divides by the sum. It produces a flat distribution where most options end up near 10–15%. Softmax runs the same logits through the exponent first. Drag the leader's logit and watch the two methods disagree.

Softmax also has cleaner derivatives, which matters because backpropagation differentiates everything in the network — including this final step. Inside attention (chapter 17's topic) the same operation runs on whole matrices, one softmax per row, each row summing to 100%.

§3The code

One built-in call, or three lines by hand. This is also where chapter 1 connects: softmax produces the distribution, and temperature, top-K, and top-P then decide how to pick from it.

probs = torch.softmax(logits, dim=-1)   # the built-in

# equivalent calculation:
exps  = torch.exp(logits)
probs = exps / exps.sum()

# then chapter 1 takes over: temperature, top-K, top-P, torch.multinomial

§4Test yourself

Nine questions on softmax.

§5Key takeaways

§1Character, word, or something in between

The two obvious designs sit at opposite ends. One id per character keeps the vocabulary tiny, but a 100,000-word sample of the FineWeb-Edu dataset becomes 612,000 tokens — six tokens per word, so a fixed context window holds six times less text, and the network's first layers spend themselves reassembling characters into words. One id per word drops the count to 100,000 tokens, but the vocabulary balloons to roughly 22,000 entries, and every entry needs a row in the embedding matrix — a matrix that already takes 31% of GPT-2 small's weights. Byte-pair encoding balances those costs. Click a row to compare the three designs.

§2How BPE builds a vocabulary

Byte-pair encoding starts from single characters and repeatedly merges the most frequent adjacent pair, growing subwords and eventually whole words. It stops at the merge budget, or earlier if no pair repeats anymore. Type your own text and drag the merge count — merged tokens turn blue, and a token that has grown into a complete word turns amber.

§3The tokenizer the model will use

OpenAI published GPT-2's trained BPE tokenizer in the tiktoken library, with 50,257 vocabulary entries. The workshop model uses it as-is and pads the table to 50,304 — a multiple of 64, which Karpathy measured at roughly a 25% training speedup because GPU matrix units prefer those shapes.

import tiktoken

tok = tiktoken.get_encoding("gpt2")
ids = tok.encode("Hello world")   # [15496, 995]
tok.decode(ids)                     # back to the string

Four special ids sit alongside the learned vocabulary. They mark structure rather than language, and the model learns what each one means during training.

§4Test yourself

Ten questions on tokenizers.

§5Key takeaways

§1The lookup

Two matrices meet here. The embedding matrix has one row per vocabulary entry and one column per hidden dimension — a dictionary that knows tokens but nothing about your sentence. The input-embeddings matrix has one row per position in your sentence, filled by copying each token's dictionary row. Click a token below. The values are stand-ins for the numbers a trained model would have learned; the shape and the copying are the real mechanics.

§2Analogy vectors

Compress GPT-2's 768-dimensional embeddings to two dimensions with PCA and the arrow from woman to queen runs roughly parallel to the arrow from man to king. The same holds for capitals and their countries. In the full matrix the relation is measurable: the nearest token to queen by cosine similarity, ignoring respellings, is king.

§3What the embedding matrix costs

Two hyperparameters set its size. Vocabulary size fixes the rows and hidden size fixes the columns. Their product is the parameter count before a single transformer block exists. At GPT-2 small's sizes — 50,257 × 768 — that is 38.6M parameters, 31% of the model's 124M total. Drag the sliders to see how quickly the cost grows.

Two more hyperparameters matter here even though they don't size this matrix. Context size is the maximum number of tokens the model accepts at once — 1,024 for GPT-2, around 500–750 words. Batch size is how many sequences run in parallel during training or inference. And at the far end of the model, the unembedding matrix maps vectors back to token scores; GPT-2 ties it to the embedding matrix by transposing it, so those 38.6M parameters get used twice.

§4Position has to be added on

The lookup in §1 copies the same row for a token wherever it appears, so "man bites dog" and "dog bites man" produce the same set of vectors. The fix is to add a small position-dependent value to every cell of the input embeddings. Sinusoidal encoding computes those values from sine and cosine waves at different frequencies — nothing to learn, and the pattern is plainly visible. GPT-2 instead learned a second matrix of position values by backpropagation, which costs memory and compute and produces a pattern that is hard to read. Toggle between them.

Modern models mostly skip both. RoPE — rotary position embedding — rotates pairs of values inside the attention step by an angle that depends on the distance between two tokens, so the model sees relative offsets ("man is one position after bites") instead of absolute slot numbers. Absolute positions can't stretch past the context length the model trained on; relative ones can. Chapter 18 shows how completely RoPE has taken over.

§5The code

In GPT-2's published weights the embedding matrix lives in wte. Indexing it with token ids is the entire lookup.

wte = model.transformer.wte          # [50257, 768]

ids = tok.encode("Hello world, I'm here")
x   = wte(torch.tensor(ids))         # input embeddings, one row per token

# "world" is id 995; its row starts ≈ -0.15 — the same numbers
# wherever the word appears, in any sentence

§6Test yourself

Ten questions on embeddings.

§7Key takeaways

§1The five steps

Three learned weight matrices turn the input into Q (query), K (key), and V (value) using three matrix multiplications. Then multiply Q by Kᵀ to score every token against every other, mask out the future, scale down by √d, softmax each row into weights, and multiply by V. Step through it on a four-token sentence, and click a row label to follow one token. I chose the starting scores. Everything after step 1 is computed from them.

§2What heads teach themselves

GPT-2 small runs 12 heads per layer across 12 layers. Inspection with a tool such as BertViz shows some heads following human-readable patterns. These patterns were learned from the training data rather than programmed as rules. The patterns below redraw four documented GPT-2 heads on the standard test sentence.

§3Sharing keys and values

In classic multi-head attention, every head has its own K and V matrices. Alternatives share them to reduce memory use. Move the slider from 12 KV sets to 1 to see the difference. DeepSeek reported about a two-percentage-point quality reduction when sharing KV across groups, alongside a large memory reduction.

§4The code

One head is a handful of lines. Heads run at a reduced width — hidden size divided by head count, so 768 ÷ 12 = 64 for GPT-2 small — then concatenate, and a learned W_O projects the result back to 768. Attention layers run without biases.

scores = q @ k.transpose(-2, -1) / math.sqrt(head_dim)
scores = scores.masked_fill(self.tril == 0, float("-inf"))
w      = torch.softmax(scores, dim=-1)
out    = w @ v            # one head; head_dim = 768 // 12 = 64

# run 12 heads, torch.cat the outputs, project with W_O
# nn.Linear(..., bias=False) throughout — no biases in attention

§5Test yourself

Ten questions on attention.

§6Key takeaways

§1The full pipeline

Click any stage to see what it does, the shape of the data leaving it, and which chapter of this lab covers it. Indented stages are inside the transformer block, which repeats 12 times in GPT-2 small.

§2Reading a modern diagram

GPT-2 XL (2019) and Qwen3-4B (2025) use the same broad structure. Qwen3 has larger dimensions and swaps four components. Click a row for the details.

Sebastian Raschka's LLM Architecture Gallery collects diagrams like these for 70+ models. Reading a mid-2026 diagram from it adds only two more unknowns on top of Qwen3 — mixture of experts, where only part of the network activates per token, and sliding-window attention, which limits some layers to a local neighborhood. Everything else on the page is chapters 1–18 of this lab.

§3The code

The block itself is two lines once its parts exist. Stack 12, and with GPT-2 small's sizes the parameter count comes out at 124M. Loading OpenAI's published GPT-2 weights into this exact structure produces logits that match Hugging Face's implementation to within noise.

def forward(self, x):
    x = x + self.attn(self.ln1(x))   # pre-norm → attention → residual add
    x = x + self.ffn(self.ln2(x))    # pre-norm → MLP → residual add
    return x

# 12 blocks + embeddings + final LayerNorm + tied unembedding = 124M params

§4Test yourself

Ten questions on transformers.

§5Key takeaways

§1The training pair

Take a window of tokens as input. The target is the same window shifted one position right — drop the first token, append the next real one. The model predicts a distribution at each position, and the loss is −log of the probability it gave the correct token. Drag the slider to see why a correct-but-unsure prediction still produces loss to learn from.

§2From 240 trillion tokens to 1.36

Common Crawl holds about 240T tokens across 300B+ downloaded pages, and almost none of it is usable as-is. FineWeb published these numbers for turning the raw crawl into the FineWeb-Edu dataset used here. Bar widths are on a log scale. Drawn linearly, the last bar would be half a percent of the first. Click a stage.

§3Quality against token count

The web is HTML, and pulling clean text out of it is imprecise. Running three extractors over the same Wikipedia page and scoring each result with two quality classifiers — DCLM's and FineWeb-Edu's 1-to-5 scale — shows the trade-off directly. The more aggressively you strip, the higher the quality scores and the fewer tokens survive. Click an extractor.

§4Sizing the run

Scaling laws make the budget predictable. Chinchilla put the optimal ratio at 20 tokens per parameter; Llama 3 landed near 40, and mixture-of-experts models report 96 to 192. Pick a model size and a ratio. The workshop model trains 124M parameters on 10B FineWeb-Edu tokens — an 80:1 ratio, well past any of these.

The other budget is GPU memory. For this model the parameters are only about 1GB — the activations dominate, because every forward value must be kept for backpropagation and they scale with batch size. Numbers below follow the VRAM calculator's estimate for this exact configuration with the AdamW optimizer.

§5Test yourself

Ten questions on pre-training.

§6Key takeaways

§1Four easy answers

There are four common ways to answer "Which model is best?" Each measures something different and has a known weakness. Click through them.

Multiple-choice sets sometimes ship with wrong answer keys, and their answers leak into training data because the internet is both the benchmark and the corpus. Answer format also changes scores. "3", "the third one" and "blue" can all express the same answer. LLM judges favor models from their own family, and automated verifiers carry their own error rates. And benchmark interactions are so short and artificial that models can tell when they're being evaluated.

§2Why our model can't take the good tests

MMLU is the readable benchmark — high-school-to-expert multiple choice a person can struggle through alongside the model. But some capabilities are emergent: below a scale threshold a model scores at the random-guess baseline, and above it the score climbs (documented by Wei's 2022 paper on emergent abilities). Drag the model size. I drew the curve as an illustration; the threshold behavior itself is the measured finding.

§3LAMBADA, and scoring the workshop model

LAMBADA avoids this scale problem by asking the model to predict the final word of a long passage. This is the same as the pre-training objective, so it works at any scale. Scoring 100 test examples by hand puts the workshop model next to the real GPT-2 small.

§4Perplexity, the readable loss

Cross-entropy has been the loss since chapter 7, but "loss = 3" is hard to interpret. Perplexity is 2 to the power of the cross-entropy. It can be read as "the model is choosing uniformly among this many tokens." Drag the loss.

§5Test yourself

Ten questions on evaluation.

§6Key takeaways

§1Before and after

These are the workshop model's real outputs, before and after tuning on the Alpaca dataset. Pick a prompt and toggle.

§2Where this sits in training

Four training phases, one loss function until the last. What changes between them is the data and, more importantly, the signal. Click a phase.

Instruction tuning mostly changes behavior rather than adding domain knowledge. It can teach the model to use a doctor's response format without teaching it medicine. The goal is a correct attempt at the task. A base model doesn't even attempt. Since 2024 the framing shifted from task verbs (summarize, rewrite, list) to capabilities: datasets exist per skill — conversation, math, coding, tool use, respecting system prompts, reasoning inside think-tags, multi-turn dialogue.

§3Chat formats

The examples have to be serialized somehow, and every model family invented its own wrapper. The differences are cosmetic. The workshop uses Alpaca — simple and widely supported.

§4LoRA, and the forgetting problem

Full fine-tuning updates every parameter, so the model, gradients and optimizer state all occupy GPU memory. LoRA freezes the base model and trains small adapters attached to a few layers, roughly 1–5% of the parameters, and performs at the same level empirically. QLoRA stores the adapter at lower numeric precision to shrink even that. Click a method.

The risk in all of it is catastrophic forgetting. Train too long on a narrow task and the model's pre-trained abilities erode — one 2024 study ran about 16 epochs of code fine-tuning and watched common-sense scores collapse. LoRA softened the damage because the base weights never move. The other defenses are running fewer steps and mixing general text back into the tuning data. The workshop run capped itself at 1,200 steps for exactly this reason, and its Balanced COPA score went from 55% to 57% — a real gain, since 50% is random with two choices and 124M parameters sits low on the common-sense emergence curve.

§5Test yourself

Ten questions on instruction tuning.

§6Key takeaways

§1Chosen and rejected

RL data contains preference pairs: two responses to one prompt, with one preferred. Pick a scenario. Each pair carries exactly one bit of supervision, and the model has to guess which of many differences that bit was about. Karpathy's line: RL is like sucking supervision through a straw.

Preference pairs can come from several sources. RLHF uses human rankings and became prominent with InstructGPT in 2022, with its separately trained reward model and frozen reference copies. Verifiable domains skip the human: RLVR rewards code that passes unit tests and math that reaches the right answer. A looser definition includes any task with an answer key. Many current datasets use stronger LLMs to write the preferences, called RLAIF. Constitutional AI uses a written constitution generates the preferences.

§2SimPO, computed live

Run the prompt with both answers through the model and collect each token's probability. A sequence's reward is the average of the log of those probabilities. The margin is β × (chosen reward − rejected reward) − γ, and the loss is −log(sigmoid(margin)). Backprop on that and cross-entropy is gone. Drag the sliders — the probabilities stand in for what a real model would assign.

§3From SimPO to the method zoo

Policy-optimization methods add different controls and infrastructure around the same parameter update. Click a method to compare SimPO, DPO, GRPO and PPO.

§4What 4,000 steps bought

The workshop run trained SimPO on Balanced COPA's 1,000 training pairs — common-sense questions with one sensible and one nonsensical cause — and scored the held-out 500-example test set as it went.

§5Test yourself

Ten questions on reinforcement learning.

§6Key takeaways

§1The technique map

Pick a category, then click a technique for a short explanation. A sensible order is mixture of experts and flash attention first, followed by quantization and KV caching for inference.

§2The last script decisions

Several of these techniques appear as settings in the workshop scripts. The largest measured difference comes from numeric precision. An A100 runs FP32 math at 19.5 teraflops and TF32 at 156 teraflops, roughly 8× the throughput on the same GPU.

§3What the 23 chapters cover

The chapters cover the path from sampling controls to pre-training, instruction tuning and RL.

§4Test yourself

Ten questions on the frontier toolbox.