A pretrained model is a cook who can make anything but has no manners, no taste and no rigor. Three classes finish it: copying worked examples (SFT), comparing two plates (DPO) and sitting a marked exam (GRPO). About 90 minutes of play.
After pretraining, a model is a young cook who can continue any recipe but rambles, ignores the order and has no taste. That's a base model. The finishing school has three classes, one per floor, and each one teaches from a different kind of signal. Every term with a wavy red underline is tappable: hover or tap it for the plain meaning and its school equivalent.
Covers smol course Units 1–2 · overview · HF LLM Course ch. 11–12 · intros · RLHF Book ch. 3 (the training overview)
Tap a signal card, then tap the floor that learns from it. (On a computer you can also drag.) The right floor lights up.
The same cook, after each class. These outputs are written to show the typical behaviour, not generated by a real model.
A model never sees "messages". It sees one long string of tokens. The chat template is the printed order ticket that turns a conversation into that string, with special tokens as the box borders. Each model family prints its own ticket. Fill one in and watch it render.
Covers smol Unit 1 · Chat Templates · HF LLM Course 11.2 · RLHF Book ch. 4 · SmolLM3-3B model card
/no_think, but the code passes enable_thinking=True. Does SmolLM3 think?Reasoning Mode: /no_think and ends the ticket with an empty <think></think> box, so the model sees "thinking already happened" and answers directly. Try it above: add /no_think, leave the switch on.<|eot_id|> and a role sits between header markers. At serving time it sees <|im_start|> markers it never learned to read, so it runs on past the end of its turn, mixes up roles or ignores the system box. Nothing crashes, which is why this bug is easy to miss. Switch the dialect above to compare the two tickets.SFT uses the same loss as pretraining: for each token, how surprised the model was by the right next token (−ln p), averaged. The only new decision is which tokens get graded. With the loss mask on, the teacher's red pen skips the customer's order the cook copied down and grades only the answer. Tap a token to change how likely the cook found it.
Covers smol Unit 1 · Supervised Fine-Tuning · HF LLM Course 11.3 · RLHF Book ch. 4
A full fine-tune rewrites the whole recipe book, and every weight being rewritten needs its gradient and optimizer state in memory. LoRA freezes the book and adds a small pad of sticky notes (an adapter) on a few kinds of page. The pad's size is the rank r. Pick a model and a machine budget and see what fits.
Covers smol Unit 1 · LoRA and PEFT · HF LLM Course 11.4 · LoRA paper (Hu et al. 2021) · shapes from each model's config.json
In DPO a taster puts two plates side by side and points at the better one: a preference pair. There's no scorecard. The reward is implicit: β × how much more likely the cook now makes a plate than the day-one photo would have. Each input below is a sequence log-probability (the sum over the answer's tokens, so a negative number).
Covers smol Unit 2 · Direct Preference Optimization · RLHF Book ch. 5 & 8 · DPO paper (Rafailov et al. 2023)
The older route to preferences trains a separate reward model: a judge who gives every plate a score. It learns from the same pairs with the Bradley–Terry loss: the chance the chosen plate wins is σ(scorechosen − scorerejected), and the loss is −ln of that chance. Slide the two scores.
TRL logs two numbers for a DPO batch: reward accuracy (the share of pairs where the chosen plate has the higher implicit reward) and the mean margin. Press the step button to nudge every pair. Pairs that are still wrong push hardest.
Every method that measures drift needs the reference model's opinion of each answer, and every opinion is a forward pass. Count the footsteps each training example costs, and how many copies of the model sit in memory.
Covers smol Unit 2 · DPO Hands-on · RLHF Book ch. 8 & 15 (regularization) · TRL docs
ref_adapter_name), otherwise peeling the notes gives you the base model, not the SFT cook.precompute_ref_log_probs does the caching.In GRPO the cook plates a group of G answers to the same exam question. A checking machine stamps each one: 1 for the right number, a small extra stamp for neat format. Then each plate is graded against the group's own average, its advantage. There's no second examiner (critic) on the payroll. Scoring with a checker makes this RLVR, the recipe behind reasoning models such as DeepSeek-R1. GRPO itself takes any score per answer, so the stamp could also come from a learned reward model, as in DeepSeekMath, which introduced GRPO.
Covers HF LLM Course 12.3–12.4 · RLHF Book ch. 6–7 · DeepSeekMath (Shao et al. 2024)
<think>, then the answer in <answer>. (Right answer: 14.)Two exam questions, four plates each. On the first, every plate is right and one is slightly messier: rewards 1, 1, 1, 0.9. On the second, half are right: 1, 0, 1, 0. Watch each question's lowest plate (outlined) as you flip "Divide by the group's spread?" above. Dividing by the spread makes a trivial 0.1 gap push harder than a real right/wrong gap. That's the difficulty/noise bias Dr. GRPO removes.
scale_rewards="none" plus loss_type="dr_grpo". TRL's current defaults are scale_rewards="group" and loss_type="dapo". This page shows only the std half.Each plate's probability under the policy has moved from πold (the policy that sampled it) to πnew. The clip caps the ratio at 1 ± ε inside the objective: the term is min(ratio × A, clipped ratio × A). When the clip changes the term, the "clip bit" lights. Plates o1–o4 of the current group; load the worked example for its numbers.
num_iterations > 1, or several optimizer steps per generation); the numbers here show a reused batch. On TRL's default num_iterations=1, πold = π, the ratio is 1 and the clip never bites.num_iterations=1 each generated batch is used for one update, so πold = π, the ratio is 1 and the clip never bites.No critic: the group is the baseline.
Three clients come to hire a graduate. For each, pick the class, the model, LoRA or full, and the machine, then grade the plan. It's scored with the same math as steps 1–6: is it the right class for the signal the client already has, is there enough data, does it fit the machine, and is it within budget? Memory is estimated at r = 16, sequence 1,024, batch 1, with checkpointing.
Covers smol Units 1–2 · HF LLM Course ch. 11–12 · RLHF Book ch. 4, 6–8
Eight questions, each answered by setting up a widget in steps 2–6 the way the question says and reading off the result. Graded instantly, with rounding tolerance.
Hire all three clients in Graduation and score 6 or better here, and the school issues your diploma.
This certifies that the bearer has finished a raw model by hand: copied the master's plates in the copying class (SFT), judged two plates in the tasting room (DPO), marked a group of answers against their own average in the exam hall (GRPO), and hired out three graduates on the right signal, within memory and within budget.
Exact: the cross-entropy on the probabilities shown; the parameter and LoRA counts, from each model's config.json on the Hub; the 16-bytes-per-parameter accounting (bf16 weights 2 + bf16 gradients 2 + fp32 master 4 + AdamW m and v 8); the DPO loss, implicit reward, margin and sigmoid; the GRPO advantages (sample std + 1e-4, as TRL computes them) and the clipped term; the pass counts per example.
Teaching models: the token probabilities in step 2 and the log-probs in step 4 are toy numbers; activation memory is a rough allowance, not a measurement; the "training step" in step 4 is an animation, not an optimizer; the four cooks' outputs in step 0 are illustrative, not generated; the πold and πnew values in step 6 are toy numbers for a reused batch; the costs in step 5 are pass counts read off the losses, not timings. The machines' "usable" memory leaves headroom for the framework (and, on a Mac, macOS's default GPU memory cap). GB means 10⁹ bytes.
The formulas are real; the example numbers are chosen to teach, not measured.
enable_thinking, the /think and /no_think flags)grpo_config.py (GRPOConfig defaults: beta=0.0, epsilon=0.2, num_iterations=1, scale_rewards="group", loss_type="dapo"; DPOConfig: β defaults to 0.1)The sibling lab, Inference Kitchen, covers what happens after training: how a finished model is served.