A student driver learns to park by reinforcement. Every lesson costs minutes of lot time, the examiner only ever sees photos, and the instructor has her own brake and wheel. This lab walks the real-world RL part of Hugging Face's Robot Learning tutorial (§3) and HIL-SERL, with soft actor-critic, RLPD and human interventions running live on a toy lot. About 70 minutes of play.
Busytown's driving school teaches parking on a small lot behind the office. The student learns the way a robot does in reinforcement learning: it tries a move, sees what happens, and is told at the end whether it parked. Its habits, which move to make from where, are the policy. In Policy Pond the duckling learned in a simulator, where a million tries are free. Here every try happens on the real lot, and the clock is running. Every term with a wavy red underline is tappable.
Covers Robot Learning tutorial §3: RL for robotics, and why real-world RL is hard (safety, resets, sample efficiency) · HIL-SERL (Luo et al. 2024) §3.1: a sparse, success-only reward
The untrained student picks one of three moves at random every second: steer left, straight on, or steer right, rolling one square forward each time. The lesson ends when it parks, hits something, pulls into the wrong bay, or after 30 moves. A reset costs 10 seconds (the instructor walks the car back) or 45 after a crash (cones, a check of the bumper).
Each move is written down as one line: where the car was (its square and which way it pointed), the move, the reward, where it ended up, and whether the lesson ended. That (s, a, r, s′, done) line is a transition; the reward is 1 only for parking straight in the exam bay, 0 for everything else, a sparse reward.
Policy Pond's learner, REINFORCE, is on-policy: it learns from a lesson driven with its current habits, makes one update, and throws the lesson away, because after the update the lesson no longer shows how the student drives. Soft actor-critic (SAC) is off-policy: it writes every move into the logbook and, after every move, studies a few random batches of old pages. Pages written by an older, clumsier student are still true facts about the lot. The number of study batches per move is the update-to-data ratio.
To give REINFORCE a fair chance, this step uses an easier practice task: drive into the painted practice box (any square, any heading), judged by the instructor's eye. Both learners see exactly the same lessons' starting squares, seed for seed.
Covers Robot Learning tutorial §3: "Off-policy algorithms like Soft Actor-Critic (SAC) tend to be more sample efficient than their on-policy counterpart … due to the presence of a replay buffer" · SAC (Haarnoja et al. 2018): a Minitaur learned to walk in about 2 hours of real time · LeRobot: utd_ratio
utd_ratio); RLPD reports a big gain from 10 on a pixel-based task.SAC keeps three things. The critic, Q(s, a): how good is this move from here, counting everything that tends to follow. The actor, π(a | s): the student's habits, pulled towards the moves the critic likes. And the temperature α: how much the student is paid for keeping its options open. The critic learns from one page at a time with a Bellman target, read off a slow copy of itself, the target network.
The moves here are three choices, so this is the discrete form of SAC (Christodoulou 2019): the averages over the next move are exact sums, not samples. The critic and the actor are tables, one number per square, heading and move; real SAC uses neural networks.
Covers Robot Learning tutorial §3, eqs. 15–17: the maximum-entropy objective and the soft TD target · HIL-SERL eqs. 1–2 · SAC (Haarnoja et al. 2018) eq. 18: automatic temperature · SAC-Discrete eqs. 10–12
A student that has had 60 lessons on the practice box (251 moves in its logbook, 888 study steps so far). Pick a page and watch the target being built.
For a fixed critic, the actor's best habits are π(a | s) ∝ exp(Q(s, a) / α): with the temperature high, all three moves look alike; with it low, the best move wins outright. The entropy measures how spread out the habits are (ln 3 = 1.10 nats for a coin with three faces). Below, three moves whose values are 0.62, 0.70 and 0.55.
SAC doesn't fix α by hand: it tunes it so the habits keep a target entropy H̄ (here 0.15 nats, a choice made for this small lot: SAC-Discrete's paper uses 0.98·ln 3 ≈ 1.08 for three moves, 98% of the maximum: on this small lot that is nearly random driving), raising α when the student gets too sure and lowering it when it's too random (eq. 18). LeRobot's code starts α at 1.0 (temperature_init); its HIL-SERL guide suggests 0.01, warning that too high a temperature "can make human interventions ineffective". Run eight students on the practice box at each setting.
In a simulator you can compute "parked" from the car's exact pose. On a real robot nothing computes it, and nobody wants to press a button every lesson for two hours. HIL-SERL trains a reward classifier: show it camera images labelled success or not, and let it pay the reward. Our examiner sits in the office and never watches the drive. Every second she gets one 5 × 4-pixel dashcam photo and ticks PASS if her score is over a threshold. A PASS ends the lesson. She is a logistic regression on the 20 pixels, trained the way the paper describes: drive the task by hand a few times, label the frames where the car is parked as positive and every other frame as negative.
Covers HIL-SERL §3.5: "200 positive data points and 1000 negative data points … around 5 minutes"; "we may collect extra data to address any false negative and false positive issues" · LeRobot: success_threshold 0.5, terminate_on_success · Reward Arcade, step 3: an optimizer farms a careless reward
Her 133 training photos: 40 of the car parked straight in the exam bay, and 93 frames from 10 of the instructor's drives. Photos are noisy: each pixel jitters, and the light changes from shot to shot. The exam bay has a chequered sign; the two staff bays have plain dark signs, and on the lot they're on the way.
Eight students, 150 lessons each, with the instructor's 20 demo drives in their logbooks (step 4), and the examiner's PASS as the only reward. The exam at the end is graded twice: by her, and by the instructor's eye.
Before the first lesson, the instructor drives the course herself, 20 times from 20 random starts, and every move goes into a second logbook, the demo buffer. She's good but human: her drives average 9.4 moves where 8.2 would do. HIL-SERL builds on RLPD: every study batch is drawn half from the demo logbook and half from the lesson log, symmetric sampling. Her pages carry rewards, so from the first study step the critic learns which squares lead to the bay, and the student drives towards them.
Covers RLPD (Ball et al. 2023) §4.1: "for each batch we sample 50% of the data from our replay buffer, and the remaining 50% from the offline data buffer" · HIL-SERL §3.2: a demo buffer "usually on the range of 20-30" · LeRobot: online_ratio 0.5 · Robot Learning tutorial §3
Driving-school cars have a second brake and wheel on the passenger side. HIL-SERL's instructor holds a SpaceMouse (in LeRobot, a gamepad or an SO-101 leader arm). When the student is about to fail, crash or pull into the wrong bay, she takes over, drives a few moves, and hands back. Her moves are interventions: they go into both logbooks, so they're in half of every batch; the student's own moves only go into the lesson log. The intervention rate is her share of the moves, averaged over 20 lessons, and it should fall to zero as the student improves.
Covers HIL-SERL §3.4: "We store the intervention data in both the demonstration and RL data buffers"; "the policy improves faster when the human operator issues specific corrections while letting the robot explore on its own otherwise" · §3.5: avoid "persistently providing long sparse interventions that lead to task successes" · LeRobot HIL-SERL guide: "Avoid intervening for long periods of time" · Robot Learning tutorial §3, Fig. 16
Simulators count episodes; real labs count hours. A crash on a real robot is a reset at best and a broken part at worst, so HIL-SERL works inside a box: in LeRobot you drive the arm around the space that solves the task, record the end-effector's minimum and maximum, and every action is clipped to stay inside (workspace bounds). Our lot gets a low fence inside the hedge: a move that would leave the lot leaves the car where it is, turned, a bump instead of a crash. Cones and parked cars are inside the box, so the fence can't help with those.
Covers Robot Learning tutorial §3: real-world RL needs "external safeguards" and manual resets are "time-consuming and error-prone" · HIL-SERL §3.3: "an impedance controller with reference limiting"; §4.3: training time "includes all scripted motion, policy rollouts, intended stops" · LeRobot: end_effector_bounds, EEBoundsAndSafety, reset_time_s
reset_time_s to 5 seconds.Three training runs went wrong. For each: run it as configured, pick a fix, re-run eight students, and name the cause. The tempting wrong fixes are on the list, and they fail for the reason the step taught. Each case is scored on the re-run and the cause.
Ten questions. Answer each by operating the widget in the step named, not from memory. Eight or more earns the check.
tests/check.js.Sources (exact quotes and sections in SOURCES.md): Luo, Xu, Wu & Levine, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning (HIL-SERL, 2024) · Capuano et al., Robot Learning: A Tutorial (2025), §3 · the LeRobot HIL-SERL guide and code · Haarnoja et al., Soft Actor-Critic Algorithms and Applications (2018) · Christodoulou, Soft Actor-Critic for Discrete Action Settings (2019) · Ball, Smith, Kostrikov & Levine, Efficient Online Reinforcement Learning with Offline Data (RLPD, 2023).
Part of a series of hands-on labs: Inference Kitchen (serving) · Finishing School (post-training) · Arm Playground (kinematics and control) · Policy Pond (reinforcement learning) · Reward Arcade (environments and evals) · Fork in the Road (diffusion policies) · Duck's-Eye View (robot vision) · Demos to Deployment (imitation learning) · Dual Controls (real-world RL with a human in the loop)