A delivery robot learns from a hundred human drives around a pretzel cart. Half went left, half went right. A model that averages them drives straight into the cart. This lab fogs the route map, lifts the fog in passes, and drives the robot with the result: the real DDPM and DDIM math from Unit 1 of the Hugging Face Diffusion Models Course, pointed at the reason robotics cares. About 75 minutes of play.
A courier robot has to get from the depot to the bakery. A pretzel cart sits in the road. The dispatcher drove the courier by joystick a hundred times and logged every drive: fifty went left around the cart, fifty went right, none through it. Those logs are the demonstrations, and the question of this lab is what to learn from them. Every term with a wavy red underline is tappable.
Covers HF Diffusion Course, Unit 1 overview: what a generative model is for · Diffusion Policy §4.1: multi-modal action distributions
Each drive is logged as a route sheet. Log a few by hand, then dump the whole book. Every dot on the map is one logged position from one drive, coloured by which way that drive went.
The simplest way to learn from the logs is behaviour cloning: fit a model that maps "where I am" to "where I go next", scored by mean squared error. Squared error has a fixed answer to an ambiguous question: the average of the plausible answers. Ask the clerk what the route is, and he averages every sheet in the book.
Diffusion's trick is to make a hard problem (draw a route) out of a thousand easy ones (make this slightly foggy map slightly clearer). To do that you first need fog you can control. The forward process mixes each logged position with Gaussian noise by an amount set by the timestep t, from 0 (clean) to 999 (pure fog). No learning happens here: it's arithmetic.
Covers notebook 02, "The corruption process" and notebook 01, step 3: DDPMScheduler, add_noise, the linear and cosine schedules
DDPM adds a little noise at each of 1,000 steps (variance βt, so a scale of √βt; the step keeps √(1−βt) of what it had). You never apply it a thousand times: ᾱt = ∏(1 − βi) gives a shortcut straight to any t. Two knobs: √ᾱ is the map's volume and √(1−ᾱ) the fog's; their squares always sum to 1, so the fog replaces the map instead of piling on top of it.
Linear is diffusers' default (β from 0.0001 to 0.02): half fog by t ≈ 260. Cosine (squaredcos_cap_v2) fogs slowly at first: half fog by t ≈ 500. Notebook 01 trains with cosine, and so does Diffusion Policy.
Now the learned half. The denoiser is a network (a UNet in the course, a 1-D one in Diffusion Policy) trained on millions of (foggy map, true map) pairs at every fog level, scored by squared error. Here she is the cartographer, and here you can watch her best single guess: for every foggy dot, the clean guess x̂₀ she would draw. Under the hood this lab uses the exact best guess for the toy data, which is what unlimited demonstrations and training converge to.
Covers notebook 02, "The model" and "Training the network": the blur at high noise · notebook 02, "Training objective": predicting ε vs predicting x₀
The toy in notebook 02 trains the network to output the clean map. DDPM (and notebook 01) trains it to output the noise that was added instead. They're the same prediction in different clothes: knowing the foggy map, t and the noise, you can solve the forward formula for the clean map. The chalk box does that for one dot.
If one guess from pure fog is a smudge, how does diffusion ever draw a real route? By not trusting the first guess. Start from pure fog, ask the cartographer for her guess, then lift only part of the fog toward it. Now the map leans, faintly, one way. Her next guess conditions on that lean and is less averaged. Repeat: each pass commits a little more, until the map has picked a side. This is the reverse process, and the rule for how much to lift per pass is the sampler.
Covers notebook 02, "Sampling": the 5-step and 40-step loops · notebook 01, step 6: scheduler.step() · notebook 02, "Sampling" (DDPM comparison): step size and churn
600 pins from 600 different fogs, for the schedule and sampler above. Each pin is one sampled position, where a route passes the cart; the wispy lines are the sampler's path through the fog, not the courier driving. Where variety comes from: the starting fog. Different fog, different side.
DDIMScheduler defaults to "leading", 900, 800, … 0, which never starts a chain at pure fog, so under it one pass wouldn't be the clerk.An image generator can take a second to draw a butterfly. A robot can't afford to: while the card is being drawn the courier stands still, and the lookout's report goes stale. Every pass is one run of the network, so passes cost time, and the time that matters is measured against the card itself: 8 pins at 10 pins a second (the paper's Ta = 8 at 10 Hz) is 0.8 s of driving. DDIM decouples training from inference: Diffusion Policy trains at 100 steps and samples in 10, and reports 0.1 s per card on an RTX 3080, an eighth of the card, which is where the 10 ms per pass below comes from. Sampling every step of this course's 1,000-step schedule would take 10 s: twelve cards' worth of driving spent standing still, on a report twelve cards old. The paper found position-control policies keep peak performance with latency of up to 4 steps.
Covers notebook 02, "Comparison to DDPM · Sampling": step size and fewer steps · Diffusion Policy §3.4: accelerating inference for real-time control
So far the cart never moved, so "draw a route" meant the same thing every day. A robot's world isn't like that. Diffusion Policy conditions the denoiser on the observation: the last two camera frames, each encoded by a ResNet-18, plus the robot's end-effector pose. Its CNN variant injects that into every layer of the network with FiLM; its transformer variant feeds it in as tokens. In the square, that's the lookout's report: "cart at +0.7 today." With it on her desk, the cartographer draws routes around where the cart is. Blindfolded, she draws them around where it used to be.
Covers Diffusion Policy §3.1: FiLM conditioning on Oₜ · contrast with Unit 2, notebook 02: class labels concatenated as input channels
Everything so far, put on the road. The cartographer no longer draws one dot: she draws a card of 16 pins (the paper's Tp = 16), the courier drives the first 8 (Ta = 8), then the lookout sends a fresh report and she draws a new card. Predict a short horizon, execute the first part, re-plan: receding-horizon control, the model-predictive-control idea running on a diffusion sampler. Every card here is denoised from fresh fog with the passes you pick: DDIM at η = 0, or Euler strides if you choose arrows, which step 7 explains. Benchmarked across 15 tasks from 4 suites, the paper's policy beat the prior state of the art by 46.9% on average. The newer action heads in π0 and GR00T N1 swap diffusion for flow matching, a relative with a straight-line target and a few Euler strides; step 7 shows it's the same picture.
Covers Diffusion Policy §3: To = 2, Tp = 16, Ta = 8, DDIM at 10 steps · §4.3: the trade-off in Ta · model-predictive control's receding horizon
When the next card is not ready in time (a 450M-parameter VLA thinking for longer than the 8 pins last), the robot stalls, swaps plans mid-swerve, or switches sides at the fork. Demos to Deployment runs every fix side by side: stop-and-think, ACT's temporal ensembling, SmolVLA's async queue and Real-Time Chunking.
Diffusion asks the cartographer for a guess: what does the clean sheet look like? Flow matching asks her for an arrow: which way, and how fast, should this pin move to get there? The path is defined as a straight line from a fog point ε to a pin x₁, xt = (1 − t)·ε + t·x₁ for t from 0 to 1, and the network is trained to output that line's velocity, x₁ − ε. Sampling is following the arrows: start in fog, take one Euler stride along the current arrow, ask again. The action experts of π0 and GR00T N1 work this way, so this is the version of the fork you'll meet in a 2025 robot. It isn't in the course, which is why it comes last. One honesty note up front: with the exact field the arrow is computed from the same guess step 2 taught, v = (x̂₁ − x) ÷ (1 − t), so it isn't a different kind of knowledge. It's a different target to train on and a different path to follow, which changes the recipe (no noise schedule to pick, uniform strides) more than it changes the answer.
Not in the course (notebook 02 mentions a different "velocity" objective in passing) · Lipman et al., Flow Matching for Generative Modeling (2023) · π0 §IV and GR00T N1 §2.1: the action experts
Every point on the map gets an arrow: the exact best straight-line velocity, averaged over every pin the point could still become. Slide t from pure fog toward the clean map and watch where the arrows point. Arrow length shows speed, capped for legibility; the chalk box has one dot's exact numbers.
The same 600 fog points, lifted two ways: DDIM's passes on the left, flow matching's Euler strides on the right, with the same number of each. Thirty of the 600 have their path through the fog traced. Watch where each fog point ends up on the two maps.
Every cartographer so far has been the exact best guess, the answer unlimited demonstrations and training converge to, which is what makes every number in this lab trustworthy. The course's notebooks do the opposite: they train a real network and watch it be imperfect. So here is an apprentice, a small real network (an MLP with 1,474 weights) that learns the square exactly the way notebook 01 trains its UNet: foggy pin in, predict the fog, squared error, Adam (the notebook uses AdamW), repeat. Train her on the dispatcher's log, then hand her the same fog the cartographer gets and compare. Everything she draws is measured, not asserted, and she is seeded, so she trains the same way for everyone.
Covers notebook 01, step 5: the training loop · notebook 02, "Training the network": what a real model gets wrong · diffusers: clip_sample
600 fog points lifted by the exact cartographer and by the apprentice as she is right now, with the same passes, both from t = 899, one notch below pure fog (the callout below says why). The exact cartographer takes the log's share as her prior, so the two maps differ only in who drew the guesses. Her clean guess is clipped to the map at every pass, as diffusers' clip_sample does; the last count on the right says how many guesses, over all the passes, the clip caught.
Four apprentices, 10,000 steps each, trained on logs where 50%, 20%, 10% and 2% of the drives went left, then the same 600 fog points lifted with 10 passes by her and by the exact cartographer with the same share as her prior. Both start from centred fog at t = 899, which pushes a rare side up for both of them, so the fair question is not "does she hit the log's share?" but "does she draw what the exact cartographer draws through the same sampler?"
Three dispatch rules came back with incident reports. For each: run the rule as dispatched, pick a fix, re-drive twenty times, and name the cause. The tempting wrong fixes are on the list, and they fail for the reason the step taught. Each case is scored on the re-drive and the cause.
Ten questions. Answer each by operating the widget in the step named, not from memory. Eight or more earns the check.
squaredcos_cap_v2) schedules built the way diffusers builds them; the denoiser, which is the exact Bayes posterior mean E[x₀ | x_t] for the toy demonstration distribution (what a noise-prediction network converges to with unlimited demonstrations and training); the ε ↔ x̂₀ conversion; DDIM sampling with η (Song et al., 2021, eq. 12) on diffusers' "trailing" timestep spacing (999, 899, …; its DDIMScheduler defaults to "leading"); the mixture mean as the squared-error-optimal single guess; flow matching's exact marginal velocity field for the same toy data, followed with forward Euler; the straightness measure; the apprentice's gradient (hand-written backprop, checked against finite differences) and Adam; and every number quoted from the Diffusion Policy, π0 and GR00T N1 papers.Sources: the Hugging Face Diffusion Models Course, Unit 1 (both notebooks) · Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models (2020) · Song, Meng & Ermon, Denoising Diffusion Implicit Models (2021) · Nichol & Dhariwal, Improved DDPM (2021) for the cosine schedule · Lin et al., Common Diffusion Noise Schedules and Sample Steps are Flawed (2023) for the "trailing" spacing · Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) for Tp/Ta/To, DDIM at 10 steps, 0.1 s on an RTX 3080, FiLM, the square cosine schedule and the 46.9% figure · Lipman et al., Flow Matching for Generative Modeling (2023) · Liu, Gong & Liu, Flow Straight and Fast (2022) for rectified flow and reflow · Black et al., π0: A Vision-Language-Action Flow Model (2024) and NVIDIA, GR00T N1 (2025) for the flow-matching action experts, their step counts, chunk lengths and latency. Further reading: The Annotated Diffusion Model, and lerobot/diffusion_pusht for a real Diffusion Policy you can run.
Part of a series of hands-on labs: Inference Kitchen (serving) · Finishing School (post-training) · Arm Playground (kinematics and control) · Policy Pond (reinforcement learning) · Fork in the Road (diffusion policies) · Demos to Deployment (imitation learning and real-time execution)