A line-painting robot learns Main Street by watching a master painter. One pothole and it wanders off; a big brain and it stops to think with the nozzle open. Every mistake stays on the road. This lab walks the imitation-learning and real-time-execution chapters of Hugging Face's Robot Learning tutorial, ACT, SmolVLA and Real-Time Chunking, with the real algorithms on a toy street. About 75 minutes of play.
The town is repainting the centre line of Main Street with a robot. Nobody wants to program it, so the master painter drives it by joystick from the kerb, twenty times down twenty different streets, and every tick is logged: what the robot's camera read, and how the master steered. Those logs are the demonstrations. The robot's own steering rule, its policy, will be learned from them. Every term with a wavy red underline is tappable.
Covers Robot Learning tutorial §4: imitation learning, behaviour cloning · HF Robotics Course Unit 1: recording a LeRobotDataset by teleoperation
The street is drawn straightened, left to right, and stretched sideways so that centimetres show (the factor is printed in the corner of every street): the grey band is the asphalt, the dashed chalk line is the centre line, and the yellow is paint. The master's hand wobbles a little, and the logbook records that too.
The simplest way to learn from the logbook is behaviour cloning: at every tick, read the camera, find the five most similar pages in the notebook, and steer the way the master steered on them. Here the learner is a lookup, not a neural network, so you can see exactly which pages it used. A neural network trained on the same pages has no examples there either: it would fill the gap with a different guess, but it would still be a guess.
The notebook is a map of situations. Plot every page by the striper's offset from the line and its heading error, and the master's twenty runs fill a tiny patch around (0, 0): a good painter is never far off. The pothole throws the apprentice out of that patch, and from then on every lookup returns the edge of the patch. That's covariate shift: the learner's own mistakes change the questions it gets asked.
Covers Robot Learning tutorial §4: compounding errors in BC · Ross & Bagnell (2010) via Ross, Gordon & Bagnell (2011), Theorem 2.1: J(π) ≤ J(π*) + T²ε
Ross and Bagnell's bound for behaviour cloning: if the learner errs with probability ε per step on the expert's states, over a horizon of T steps it can make up to T²ε mistakes, because a mistake early on can cost every step after it. Run twelve streets at each length, with potholes at 0.08 per metre, and see how the off-line paint grows.
The fix is to get pages for the places the apprentice actually goes. DAgger does it in rounds: the apprentice drives six practice streets full of potholes (with the master grabbing the joystick back a share β of the time), the master rides along and says what he would steer at every state it reached, those pages go into the notebook, and the apprentice is refit. Round 1 was the demonstrations themselves (β₁ = 1); after that β halves each round.
Covers Ross, Gordon & Bagnell (2011), Algorithm 3.1: π_i = β_i·π* + (1 − β_i)·π̂_i, β_i = p^(i−1) · Theorem 3.2: cost linear in T
The apprentice now gets a big brain, the kind of vision-language-action model that reads cameras and instructions. A model that size can't answer fifty times a second, so it answers once with a work order: 50 waypoints, where the nozzle should be on each of the next 50 ticks. A fixed servo steers to them. π0 and SmolVLA both predict 50 actions; ACT predicts 100, and in its ablation success rose from 1% with one action at a time to 44% with chunks of 100. The catch is time. This step pulls over to think: synchronous inference.
Covers Robot Learning tutorial §4: action chunking (ACT), "Optimized Inference" · π0: H = 50 at 50 Hz, re-plan every 25 actions · RTC §4.2: 76 ms model latency, d ≈ 6 ticks with the network
Now there's wet cement on every street, and the master went round it on the left in half his runs and on the right in the other half. A work order is a sample from those runs, so it commits to one side. ACT's way to drive smoothly is temporal ensembling: ask for a new order every tick, and drive, at each tick, the weighted average of every order that has a waypoint for it, weights exp(−m·i) with i = 0 the oldest and m = 0.01 (from ACT's code). Every order votes. That smooths the seams, and it is fifty orders a second, with d of them being drafted at once while each one thinks: d times the compute of a single desk.
Covers ACT §IV-A, Algorithm 2: "we query the policy at every timestep", w_i = exp(−m·i) · LeRobot: temporal ensembling requires n_action_steps = 1 · RTC: temporal ensembling "performs poorly across the board" in simulation and couldn't run on the robots at +100 ms
SmolVLA's answer to the blobs: never pull over. Keep driving the current order and radio for the next one while there are still waypoints in hand. The rule is one number, the threshold g: send a fresh observation whenever g·50 or fewer waypoints remain. g = 0 waits for an empty queue (the paper's sequential limit, here: drive all 50, then stop); g = 1 would ask at every tick, but with one order in flight at a time it asks again the moment the last one lands, every d ticks. When the new order lands, the waypoints that overlap the old queue are merged by an aggregation rule; the numbers below use "keep the newest" unless you pick another. The queue never runs dry if g ≥ d/50.
Covers SmolVLA §3.3, Algorithm 1: |A_t|/n < g, n = 50 · §4.6: async finished tasks ~30% faster (13.75 s → 9.70 s) and placed 19 cubes in 60 s against 9 · LeRobot async: chunk_size_threshold (0.5 in code), aggregate_fn "weighted_average" = 0.3·old + 0.7·new
The plainest way to think while moving, the RTC paper's "naive async": start drafting the next order d ticks before the current one is due to run out, and switch to it the moment it lands, dropping the waypoints whose time has passed. The new order was drafted from a camera frame d ticks old, without knowing which waypoints the striper would drive in the meantime. Where the two plans disagree there's a seam; at the cement, where they can disagree about the side, there's a side flip.
Covers RTC §2: inference delay d = ⌊δ/Δt⌋, execution horizon s, d ≤ s ≤ H − d · "naively switching chunks as soon as the new one is ready"
Physical Intelligence's fix keeps the async timing and changes the drafting. By the time the new order lands, the first d of its waypoints will already have been driven from the old order: they are frozen, and the new order must continue them. The rest of the overlap is soft: the new order is pulled towards the old one with weights that decay to zero, so it eases from the old plan into the new instead of jumping. RTC does this by guiding a flow-matching sampler (inpainting), with no retraining. The notebook has no sampler to guide, so this page uses an analogue over its pages: draw the 16 most similar, shift each to continue the frozen waypoints, weight each by its agreement under the soft mask, sample one, and blend it in with the same mask. The mask is RTC's; the conditioning is a lookup-table stand-in for its guidance.
Covers RTC (Black, Galliker & Levine 2025) eq. 5: W_i = 1 for i < d; c_i(e^c_i − 1)/(e − 1), c_i = (H − s − i)/(H − s − d + 1), for d ≤ i < H − s; 0 after · robust to delays "in excess of 300 milliseconds"; 20% faster than synchronous on π0.5
Pour the cement late, 1.6 m ahead of the striper, and measure how far it drives blind: metres on orders drafted before the pour. Frozen waypoints are, by design, not up for revision.
Three stripers came back with complaints from the town. For each: run it as configured, pick a fix, re-run it on twenty streets, and name the cause. The tempting wrong fixes are on the list, and they fail for the reason the step taught. Each case is scored on the re-run and the cause.
Ten questions. Answer each by operating the widget in the step named, not from memory. Eight or more earns the check.
Sources (exact quotes and sections in SOURCES.md): Capuano et al., Robot Learning: A Tutorial (2025), §4–5 · Ross, Gordon & Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2011) · Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT, 2023) · Black et al., π0 (2024) · Shukor et al., SmolVLA (2025) and the LeRobot async inference docs · Black, Galliker & Levine, Real-Time Execution of Action Chunking Flow Policies (2025).
Part of a series of hands-on labs: Inference Kitchen (serving) · Finishing School (post-training) · Arm Playground (kinematics and control) · Policy Pond (reinforcement learning) · Fork in the Road (diffusion policies) · Duck's-Eye View (robot vision) · Demos to Deployment (imitation learning and real-time execution) · Inside a VLA (what the drafting desk is doing)