Fork in the Roada hands-on lab

Why robots use diffusion to choose actions

A delivery robot learns from a hundred human drives around a pretzel cart. Half went left, half went right. A model that averages them drives straight into the cart. This lab fogs the route map, lifts the fog in passes, and drives the robot with the result: the real DDPM and DDIM math from Unit 1 of the Hugging Face Diffusion Models Course, pointed at the reason robotics cares. About 75 minutes of play.

0 · The square: a hundred drives, one fork

A courier robot has to get from the depot to the bakery. A pretzel cart sits in the road. The dispatcher drove the courier by joystick a hundred times and logged every drive: fifty went left around the cart, fifty went right, none through it. Those logs are the demonstrations, and the question of this lab is what to learn from them. Every term with a wavy red underline is tappable.

The courier = the robot
its driving rule is the policy
The dispatcher = the human demonstrator
teleoperation, one drive at a time
Route sheets = the demonstrations
the training data of imitation learning
The pretzel cart = the obstacle
the reason the demos fork
The clerk's average = a regression policy
trained with squared error (step 0)
Fog = noise
rolled over the map on purpose (step 1)
The cartographer = the denoiser
guesses the clean map from a foggy one (step 2)
The lookout's report = the observation
what the camera sees now (step 5)
The waypoint card = an action chunk
16 pins ahead, drive 8, look again (step 6)

Covers HF Diffusion Course, Unit 1 overview: what a generative model is for · Diffusion Policy §4.1: multi-modal action distributions

Watch the dispatcher drive

Each drive is logged as a route sheet. Log a few by hand, then dump the whole book. Every dot on the map is one logged position from one drive, coloured by which way that drive went.

0
drives logged
0
went left
0
went right
went leftwent rightinside the cart

The clerk's rule: average the sheets

The simplest way to learn from the logs is behaviour cloning: fit a model that maps "where I am" to "where I go next", scored by mean squared error. Squared error has a fixed answer to an ambiguous question: the average of the plausible answers. Ask the clerk what the route is, and he averages every sheet in the book.

–
average route, x at the cart's level
–
result
average = (50 × left + 50 × right) ÷ 100
left ≈ −1.3 · right ≈ +1.3 · cart spans −0.55 to +0.55
average ≈ 0.0: straight through
This is not a weak model. It's the right answer to the wrong question: squared error asks for the single guess that is least wrong on average, and at a fork that guess is the middle. The Diffusion Policy paper calls the demonstrations' distribution multi-modal and says explicit regression policies are unsuitable for them; the clerk's average is that failure in its simplest form.
Guess first!
Average 50 left routes and 50 right routes. At the cart's level, where does the average route go?
Straight through. The average of −1.3 and +1.3 is 0.0, and the cart is parked at 0. Press the button and watch the dent.
Guess first!
Give the clerk ten times more drives: a thousand sheets, still half left and half right. Does he stop hitting the cart?
No. More sheets make the average more precise, not different: it's still 0.0. The problem is the rule, not the data. That's why the fix in this lab is a different kind of model, not a bigger dataset.
Say it out loud
The demonstrations fork. A model trained with squared error answers a fork with the average, and the average of left and right is straight through the cart.

1 · Fog rolls in: the forward process

Diffusion's trick is to make a hard problem (draw a route) out of a thousand easy ones (make this slightly foggy map slightly clearer). To do that you first need fog you can control. The forward process mixes each logged position with Gaussian noise by an amount set by the timestep t, from 0 (clean) to 999 (pure fog). No learning happens here: it's arithmetic.

Covers notebook 02, "The corruption process" and notebook 01, step 3: DDPMScheduler, add_noise, the linear and cosine schedules

Roll the fog

t: how long the fog has rolled0
Fog schedule
1.00
√ᾱ · the map's volume
0.00
√(1−ᾱ) · the fog's volume
0%
dots now on the wrong side of the road
–
t where the map is half fog
xt = √ᾱt · x0 + √(1 − ᾱt) · ε
logged on a left drivelogged on a right drive

The schedule: how fast the fog thickens

DDPM adds a little noise βt at each of 1,000 steps. You never apply it a thousand times: ᾱt = ∏(1 − βi) gives a shortcut straight to any t. Two knobs: √ᾱ is the map's volume and √(1−ᾱ) the fog's; their squares always sum to 1, so the fog replaces the map instead of piling on top of it.

Linear is diffusers' default (β from 0.0001 to 0.02): half fog by t ≈ 260. Cosine (squaredcos_cap_v2) fogs slowly at first: half fog by t ≈ 500. Notebook 01 trains with cosine, and so does Diffusion Policy.

√ᾱ, the map · √(1−ᾱ), the fog · the marker is your t
Guess first!
With the cosine schedule, at roughly which t is the map half map, half fog (√ᾱ ≈ √(1−ᾱ))?
About 500 (496 exactly). Cosine keeps the map loud for the first few hundred steps. Read it off the chart where the two curves cross.
Guess first!
Switch to the linear schedule. Does the fog reach half strength earlier or later than with cosine?
Earlier: t = 259. It matters in step 3: with few passes, where the sampler stops depends on the schedule.
Say it out loud
x_t equals √ᾱ times the map plus √(1−ᾱ) times fog: a crossfader set by the schedule. Cosine fogs slowly early; linear is half fog by about t = 260.

2 · The cartographer: one guess from the fog

Now the learned half. The denoiser is a network (a UNet in the course, a 1-D one in Diffusion Policy) trained on millions of (foggy map, true map) pairs at every fog level, scored by squared error. Here she is the cartographer, and here you can watch her best single guess: for every foggy dot, the clean guess x̂₀ she would draw. Under the hood this lab uses the exact best guess for the toy data, which is what a perfectly trained network converges to.

Covers notebook 02, "The model" and "Training the network": the blur at high noise · notebook 02, "Training objective": predicting ε vs predicting x₀

Her guess at every fog level

t: how foggy the map she's given is300
–
guesses left
–
guesses right
–
guesses inside the cart
one dot, the weight she puts on each side
the foggy dot she's givenguess, leftguess, rightguess, inside the cart
Slide t to 999. Every guess lands in the middle of the cart. From pure fog, the guess that is least wrong on average is the average of every route: notebook 02's "blurry mess close to the mean of the dataset". It is the clerk's average from step 0, produced by a different model for the same reason.

Predict the noise or predict the map?

The toy in notebook 02 trains the network to output the clean map. DDPM (and notebook 01) trains it to output the noise that was added instead. They're the same prediction in different clothes: knowing the foggy map, t and the noise, you can solve the forward formula for the clean map. The chalk box does that for one dot.

x̂0 = (xt − √(1 − ᾱt) · ε̂) ÷ √ᾱt
Why prefer the noise target? Notebook 02's answer: training samples t at random, and the two objectives weight those samples differently. Predicting the noise "puts more weight on lower noise levels", which is where the fine detail of the map gets decided. (A second intuition, not from the course: the noise target keeps the same unit scale at every t, while a clean-map target is trivial at low t and the smudge at high t.) The Diffusion Policy paper describes its network as learning "the gradient of the action score function", and says ε̂ "effectively predicts the gradient field ∇E(x)" of an energy. That gradient points from data toward fog, so every sampling pass steps against it.
Guess first!
Slide t to 999, pure fog. Where do the cartographer's guesses land?
All in the cart, 100%. Pure fog carries no hint of a side, so the squared-error-best guess is the average of everything: the middle. Notice the weights in the chalk box: 0.50 left, 0.50 right.
Guess first!
DDPM trains the network to predict the noise ε rather than the clean map. Is that a different prediction?
Same information. x̂₀ = (x_t − √(1−ᾱ)·ε̂) ÷ √ᾱ, and the chalk box recovers the guess from the noise prediction exactly. Notebook 02's reason for preferring the noise target is the implicit loss weighting: it puts more weight on the low-noise levels.
Say it out loud
From pure fog, the squared-error-best guess is the average of every route: the smudge. That's not a weak network. It's what squared error rewards, and the next step is how diffusion gets around it.

3 · Fog lifts in passes: the sampler

If one guess from pure fog is a smudge, how does diffusion ever draw a real route? By not trusting the first guess. Start from pure fog, ask the cartographer for her guess, then lift only part of the fog toward it. Now the map leans, faintly, one way. Her next guess conditions on that lean and is less averaged. Repeat: each pass commits a little more, until the map has picked a side. This is the reverse process, and the rule for how much to lift per pass is the sampler.

Covers notebook 02, "Sampling": the 5-step and 40-step loops · notebook 01, step 6: scheduler.step() · notebook 02, "Sampling" (DDPM comparison): step size and churn

Lift the fog

Passes (K)
Sampler
Fog schedule
–
went left
–
went right
–
inside the cart
still foggywent leftwent rightinside the cart

The ladder: passes against dents

600 pins from 600 different fogs, for the schedule and sampler above. Each pin is one sampled position, where a route passes the cart; the wispy lines are the sampler's path through the fog, not the courier driving. Where variety comes from: the starting fog. Different fog, different side.

Two samplers, one spacing. DDIM with η = 0 is deterministic: the same fog always gives the same pin. η = 1 blows a little fresh fog back in between passes, which is what DDPM's own step does. Both reach the same ladder here; what buys the speed is skipping timesteps, not η, and Diffusion Policy uses DDIM to plan in 10 passes. This lab visits t = 999, 899, … 99 (diffusers' "trailing" spacing). diffusers' DDIMScheduler defaults to "leading", 900, 800, … 0, which never starts a chain at pure fog, so under it one pass wouldn't be the clerk.
Guess first!
One pass from pure fog, cosine schedule. What share of the 600 pins end inside the cart?
All 600. One pass is one guess from pure fog, and step 2 showed where that lands: the average, the cart. K = 1 is the clerk.
Guess first!
Five passes, cosine, η = 0. What share end inside the cart?
About 2% (10 of 600). Two passes: 39%. Three: 11%. Ten: 1 of 600. Each pass starts from a map that already leans, so the guesses stop averaging. Run K = 2, then 3, then 5 and watch the middle empty out.
Guess first!
Switch to the linear schedule and take 2 passes. Compared with cosine's 39% in the cart, is it better or worse?
Much worse: 94%. The two-pass sampler stops at t = 999 and t = 499, and under the linear schedule t = 499 is still mostly fog (ᾱ = 0.08, so √(1−ᾱ) = 0.96), so the second guess is nearly as averaged as the first. Under cosine, t = 499 is half map. Few-pass sampling and the schedule are one decision. (Diffusion Policy reports that the square cosine schedule simply worked best on its tasks, and ties the choice to how well the policy captures the high- and low-frequency parts of the action signal.)
Say it out loud
Lift the fog in passes. Each guess starts from a map that already leans one way, so the route commits to one side instead of averaging both. Ten passes are plenty here; one is never enough.

4 · The stopwatch: how many passes can you afford

An image generator can take a second to draw a butterfly. A robot can't afford to: while the card is being drawn the courier stands still, and the lookout's report goes stale. Every pass is one run of the network, so passes cost time, and the time that matters is measured against the card itself: 8 pins at 10 pins a second (the paper's Ta = 8 at 10 Hz) is 0.8 s of driving. DDIM decouples training from inference: Diffusion Policy trains at 100 steps and samples in 10, and reports 0.1 s per card on an RTX 3080, an eighth of the card, which is where the 10 ms per pass below comes from. Sampling every step of this course's 1,000-step schedule would take 10 s: twelve cards' worth of driving spent standing still, on a report twelve cards old. The paper found position-control policies keep peak performance with latency of up to 4 steps.

Covers notebook 02, "Comparison to DDPM · Sampling": step size and fewer steps · Diffusion Policy §3.4: accelerating inference for real-time control

The budget

time per pass (ms)10
Pins per card (Ta), at 10 pins a second
Fog schedule
800 ms
one card lasts
80
passes before drawing outlasts driving
–
of each cycle spent waiting, at 10 passes
10 s
all 1,000 passes of the course's schedule
passes ≤ card length ÷ time per pass
Dents from the step 3 sampler (η = 0, 600 pins). Waiting share = drawing ÷ (drawing + driving). Rows in red take longer to draw than to drive; the last row walks the course's whole 1,000-step schedule.
The courier stands still while the card is drawn. That's this lab's model of latency, and it's a simplification: real controllers can keep executing the last card while the next one is computed. But a card drawn from a 10-second-old observation is 10 seconds stale either way, which is what the review board's third case is about.
Guess first!
10 ms per pass and a card of 8 pins (0.8 s of driving). How many passes before drawing the card takes longer than driving it?
80, and step 3's ladder says 10 already leave 1 pin in 600 in the cart. That is the whole argument for DDIM in Diffusion Policy: the passes you can afford, with the courier waiting an eighth of the time, are more than enough.
Guess first!
Walk the course's whole schedule: 1,000 passes at 10 ms each. How long does the courier wait for one card?
10 seconds, for a card that covers 0.8 seconds of driving (1.6 s if all 16 pins were driven). The square will have changed before the courier moves. The paper's own DDPM baseline trains and samples at 100 steps, which would be about 1 s on the same GPU.
Say it out loud
A card of 8 pins is 0.8 s of driving. Ten DDIM passes at 10 ms take 0.1 s, an eighth of that, and already keep the courier out of the cart. A thousand passes would take 10 s: the square has moved on.

5 · The lookout: conditioning on what the robot sees

So far the cart never moved, so "draw a route" meant the same thing every day. A robot's world isn't like that. Diffusion Policy conditions the denoiser on the observation: the last two camera frames, each encoded by a ResNet-18, plus the robot's end-effector pose. Its CNN variant injects that into every layer of the network with FiLM; its transformer variant feeds it in as tokens. In the square, that's the lookout's report: "cart at +0.7 today." With it on her desk, the cartographer draws routes around where the cart is. Blindfolded, she draws them around where it used to be.

Covers Diffusion Policy §3.1: FiLM conditioning on Oₜ · contrast with Unit 2, notebook 02: class labels concatenated as input channels

Move the cart

where the vendor parked the cart today+0.7

With the report

–
inside the cart
–
went left

Blindfolded

–
inside the cart
–
went left
How the report gets in. Unit 2's class-conditioned notebook embeds the label and concatenates it to the input as extra image channels: the condition enters once, at the door. FiLM instead lets the observation produce a scale and a shift for every channel at every layer, so "what I see" steers every stroke of "what I draw". Same purpose; the deeper injection is what makes the policy react to a camera.
Guess first!
Park the cart at +1.0 and blindfold the cartographer. Where do her routes go?
Around the old spot: her left and right routes are ±1.3 from the centre line, and the cart now spans +0.45 to +1.55, so 28% of the pins end inside it, every one from the right route. With the report, 2 of 600.
Guess first!
In Diffusion Policy, how does the camera observation reach the denoiser?
FiLM, in the CNN variant. The concatenation answer is how Unit 2's class-conditioned toy does it, which works for a label but enters only at the input. FiLM conditions every layer, and the paper uses it for both the observation and the denoising step k. (The transformer variant passes the observation in as input tokens instead.)
Say it out loud
The observation is the conditioning. The cartographer draws routes around today's cart because the report is on her desk while she draws every pin. Blindfolded, she draws around where it used to be.

6 · Pin 16, drive 8, look again: the robot drives

Everything so far, put on the road. The cartographer no longer draws one dot: she draws a card of 16 pins (the paper's Tp = 16), the courier drives the first 8 (Ta = 8), then the lookout sends a fresh report and she draws a new card. Predict a short horizon, execute the first part, re-plan: receding-horizon control, the model-predictive-control idea running on a diffusion sampler. Every card here is denoised from fresh fog with the passes you pick (DDIM, η = 0). Benchmarked across 15 tasks from 4 suites, the paper's policy beat the prior state of the art by 46.9% on average. The newer action heads in π0 and GR00T N1 swap diffusion for flow matching, a relative that needs fewer passes; the picture is the same.

Covers Diffusion Policy §3: To = 2, Tp = 16, Ta = 8, DDIM at 10 steps · §4.3: the trade-off in Ta · model-predictive control's receding horizon

Drive

Who draws the card
Passes per card
Pins driven per card (Ta)
where the cart starts0.0
Sampler: DDIM, η = 0 · 10 ms per pass · 10 pins per second
0.0 s
clock
–
card
ready
the courier is
0
dents
pin being drivendrivenplanned, then thrown awaydashed box: where the courier thinks the cart is

Twenty drives, no animation

–
reached the bakery
–
hit the cart
–
side flips per drive
–
seconds per drive
Side flips count how often a fresh card picks the other side of the cart from the last one, before the courier reaches it. Re-plan every pin and the courier dithers; drive 8 per card and it commits. That's the paper's §4.3 trade-off: Ta "encourages temporal action consistency while remaining responsive". The other end of the dial is worth a run too: drive the whole card (Ta = 16) with the push and the lookout on, and 10 of 20 drives dent the cart. The report arrives, but nobody re-reads it until the card is used up.
Guess first!
Re-plan after every pin (Ta = 1) instead of every 8. Over 20 drives, what changes?
Side flips jump from about 0.05 per drive to about 1.9: each fresh card is a fresh sample, and near the depot the fork is still 50/50, so the courier zigzags. Dents stay at 0. Driving 8 pins per card is what commits it.
Guess first!
Blindfold the cartographer (lookout off) and let the vendor push the cart at 0.5 s. Of 20 drives, how many hit the cart?
11 of 20. A blind right-side route skirts the cart's old spot at about +1.0 to +1.3, and the pushed cart now reaches +1.25, so every drive that ends up on the right clips its corner. Left-side drives stay clear. With the lookout on: 0 of 20, and the courier re-routes after the push.
Guess first!
Switch to the clerk's averager, lookout on, with the push. Does the averager still hit the cart?
No, 0 of 20. In this toy, once the cart is off to one side the demonstrators almost all went the other way, so the average is a real route. The averager fails exactly where the demonstrations fork evenly, which is the situation the paper's Push-T example illustrates. Park the cart back at 0.0 with no push and it dents every time.
Say it out loud
Pin 16, drive 8, look again. The card gives smooth, committed motion; the fresh report gives reactivity; ten DDIM passes make it fast enough to do this ten times a second.

★ Dispatch review board

Three dispatch rules came back with incident reports. For each: run the rule as dispatched, pick a fix, re-drive twenty times, and name the cause. The tempting wrong fixes are on the list, and they fail for the reason the step taught. Each case is scored on the re-drive and the cause.

☆☆☆

✓ Field test

Eight questions. Answer each by operating the widget in the step named, not from memory. Six or more earns the check.

What's exact here, and what's a teaching model

  • Exact, real math: the forward process x_t = √ᾱ·x₀ + √(1−ᾱ)·ε; the linear and cosine (squaredcos_cap_v2) schedules built the way diffusers builds them; the denoiser, which is the exact Bayes posterior mean E[x₀ | x_t] for the toy demonstration distribution (what a perfectly trained noise-prediction network converges to, with no training error); the ε ↔ x̂₀ conversion; DDIM sampling with η (Song et al., 2021, eq. 12) on diffusers' "trailing" timestep spacing (999, 899, …; its DDIMScheduler defaults to "leading"); the mixture mean as the squared-error-optimal single guess; and every number quoted from the Diffusion Policy paper.
  • Teaching models, labelled on the page: the square, the two demonstrated routes and how the demonstrators' choice of side leans with the courier's position are hand-designed; the courier teleports between checkpoints ten times a second; it stands still while a card is drawn; the vendor's push and pacing are one-number toys. No network is trained here, and nothing here predicts how a real Diffusion Policy run behaves.
  • The directions are real; the courier isn't a real robot. Real action chunks are joint positions or end-effector poses at 10–50 Hz, conditioned on camera frames, from a network trained on hundreds of demonstrations. The fork, the smudge, the passes, the budget and the re-planning are the same ideas at that scale.

Sources: the Hugging Face Diffusion Models Course, Unit 1 (both notebooks) · Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models (2020) · Song, Meng & Ermon, Denoising Diffusion Implicit Models (2021) · Nichol & Dhariwal, Improved DDPM (2021) for the cosine schedule · Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) for Tp/Ta/To, DDIM at 10 steps, 0.1 s on an RTX 3080, FiLM, the square cosine schedule and the 46.9% figure. Further reading: The Annotated Diffusion Model, and lerobot/diffusion_pusht for a real Diffusion Policy you can run.