Policy Ponda hands-on lab

How a robot learns by trial and error

A duckling learns to cross a pond the way a robot learns to walk: try something, get scored, nudge the habit. Run real policy-gradient and PPO updates in your browser, drag PPO's clip, and break a reward function on purpose. From REINFORCE to PPO in about 90 minutes of play.

0 · The pond: act, get scored, repeat

Reinforcement learning is one loop. The agent does something, the environment answers with a new situation and a reward, and the agent wants as much reward as possible over time. Here the agent is a duckling and the reward is breadcrumbs. Every term with a wavy red underline is tappable.

Duckling = the agent
its habits are the policy π
The pond = the environment
the current, the reeds, the bread boat
Breadcrumbs = reward
the only feedback it ever gets
A crossing = an episode
start bank to far bank
Mama Duck = the critic
knows what a crossing usually earns (step 2)
Lane ropes = PPO's clip
don't change too much per lesson (step 3)
The scorecard = the reward function
what you choose to pay for (step 5)
The flock = parallel environments
4,096 ducks practising at once (step 6)

Covers HF Deep RL Course, Unit 1: the RL framework, the reward hypothesis, discounting, episodic tasks

Watch one crossing

A narrow pond of 8 spots. Crumbs sit on the far bank (+20, and the crossing ends). Every paddle costs 1 crumb of effort. The current pushes the duckling the wrong way 15% of the time, and it gets 25 paddles before the day ends.

Habit: how often it tries to go right
Each step the pond hands back (where it was, what it tried, crumbs, where it ended up): exactly what a robot's training loop records.
–
paddles taken
–
crumbs, simply added up
–
return G with your γ below

γ, the discount: how far ahead does it care?

γ
A reward k steps away is worth γk of one now.
Microduck, a real 25 cm robot duck, trains with γ = 0.99 at 50 steps per second. Step 6 opens its actual config.
Bar k = how much a crumb k steps in the future counts today
steps ahead it cares (≈ 1 ÷ (1 − γ))
seconds ahead, at 50 steps/s
10 crumbs, 30 steps away, worth today
Guess first!
Offer the duckling 1 crumb right now, or 10 crumbs 30 steps from now. With γ = 0.9, which does it prefer?
1 crumb now. 10 × 0.9³⁰ = 0.42 crumbs in today's money, less than 1. At γ = 0.99 the same offer is worth 7.4, so it would wait. Set the dial to 0.9, then 0.99, and read the gold tag.
Guess first!
Microduck: γ = 0.99 at 50 steps per second. Roughly how far into the future does it care?
About 2 seconds. 1 ÷ (1 − 0.99) = 100 steps, and 100 steps at 50 per second is 2 seconds: enough to care whether this step sets up a stumble a second from now. γ = 0.9 would be 10 steps, 0.2 seconds.
Say it out loud
The duckling acts, the pond answers with a new spot and some crumbs, and it wants the most crumbs over time. γ sets how far ahead that is: 0.99 at 50 steps a second means about two seconds.

1 · Lucky crossings: nudge the habit toward what paid

Now there are three ways across. A hugs the reeds: about 1 crumb, every time. B passes the bread boat: 2 crumbs on average, but wildly up and down. C goes through the mud: about 0.3. The duckling's habit is three numbers turned into probabilities (a softmax). REINFORCE updates it after every crossing: make the route just taken more likely, in proportion to the crumbs it brought. This is the real algorithm, running in your browser.

Covers HF Deep RL Course, Unit 4: policy-based methods, the policy gradient, REINFORCE (Monte Carlo policy gradient)

Train it

Step size (learning rate α)
A · B · C: the chance of choosing each route, crossing by crossing
0
crossings so far
expected crumbs per crossing (best possible: 2.00)
–
last crossing
Guess first!
The duckling takes the muddy route C and brings back 0.3 crumbs. Plain REINFORCE. Does route C get more likely or less likely?
More likely. The update is crumbs × (push toward the route taken), and 0.3 is positive, so C gets pushed up, just less than a big haul would push. Plain REINFORCE only learns that C is bad by comparing across many crossings. Step 2 fixes this by asking "better or worse than usual?" instead of "positive or negative?". Click 1 crossing until it picks C and watch its bar.

The catch: one crossing is a very noisy clue

Freeze the habit at "no preference" (⅓ each). Each dot below is what one crossing says the push toward route B should be. They're all estimates of the same true push, the dashed line.

Guess first!
How do the 30 single-crossing estimates compare to the true push?
All over the place. On average they're right (that's what the policy gradient theorem guarantees), but any single crossing can say "B is amazing" or "B is terrible". That's variance, and it's why plain REINFORCE learns slowly and jerkily. The HF course calls this the variance problem, and it's the reason Unit 6 exists.
Say it out loud
REINFORCE makes whatever the duckling just did more likely, in proportion to the crumbs it got. It points the right way on average, but any single crossing is a very noisy clue.

2 · Mama's guess: score against what was expected

Mama Duck has watched a thousand crossings. She knows roughly what a crossing from here usually earns: that's the value V(s). Score each crossing as crumbs − Mama's guess, the advantage, instead of raw crumbs. Subtracting a baseline doesn't change which way the push points on average. It removes the part of every haul that would have happened anyway, so the noise drops.

Covers HF Deep RL Course, Unit 6: the variance problem, advantage actor-critic (A2C) · and the value functions and TD learning from Unit 2

Same 30 crossings, with and without Mama

Crumbs just for finishing
Every crossing earns this on top, whatever the route. Real tasks are full of rewards like this: "alive" bonuses, "upright" bonuses.
Without Mama (raw crumbs)
With Mama (crumbs − her guess)
spread without Mama
spread with Mama
the true push (same for both)
Guess first!
Raise "crumbs just for finishing" from 0 to 10. What happens to the spread without Mama?
It balloons. The bonus says nothing about which route is better, but plain REINFORCE multiplies it into every update, shoving whatever route was just taken upward. With Mama subtracting her guess, the bonus cancels and the spread doesn't move. Slide it and watch.

Race: 40 ducklings with Mama, 40 without

300 crossings each, with the finishing bonus you set above
without Mama · with Mama (a running average of hauls). Line = the average duckling's expected crumbs (bonus excluded); band = how much ducklings differ.
Set the bonus to 0 and the two flocks run neck and neck. That's real, not a bug: a baseline helps most when every crossing carries a large common reward, which real tasks nearly always do ("alive" bonuses, "upright" bonuses). The larger the shared part, the more the baseline removes.

One step at a time: the TD error

Mama doesn't have to wait for the far bank. After a single paddle she can compare what happened with what she expected: the reward just earned plus her (discounted) guess from the new spot, minus her guess from the old spot. That's the TD error δ, her one-step surprise. Using her own guess for the rest of the crossing is called bootstrapping.

Crumbs this step (r)
Mama's guess here, V(s)
Mama's guess at the new spot, V(s′)
Guess first!
No crumbs this step (r = 0), but the duckling moved from a spot Mama rates 5 to one she rates 6 (γ = 0.99). Good step or bad step?
Good. δ = 0 + 0.99 × 6 − 5 = +0.94. No crumbs arrived yet, but it moved somewhere better than Mama expected, and that's worth reinforcing now instead of waiting for the far bank. That's the whole trick of actor-critic.
Say it out loud
Mama's guess V(s) doesn't change which way the push points on average; it subtracts what was coming anyway, so the noise collapses. The TD error, r + γ·V(next) − V(here), scores a single step against her guess.

3 · The lane ropes: PPO

Crossings are expensive, so you'd like to learn several lessons from each batch of them. But every lesson moves the habit, and after a few the habit is no longer the one that collected the batch, so the batch starts lying. PPO measures the drift for each crossing as a ratio, r = πnew(route) ÷ πold(route), and puts lane ropes at 1 − ε and 1 + ε. Inside the ropes, learn freely. Outside, credit for going further stops, but undoing a mistake is always allowed.

Covers HF Deep RL Course, Unit 8: the intuition behind PPO, the clipped surrogate objective, visualizing the six cases

Drag the ratio across the ropes

Ratio r = new ÷ old
Was this crossing better or worse than Mama expected?
Rope width ε
Dashed = r × A with no ropes · solid = what PPO actually climbs, min(r·A, clip(r)·A) · shaded = inside the ropes
does this crossing still push the habit?
which of the six cases you're in

All six cases (the course's table, live)

Each tile is one case, drawn with your ε. Tiles you've visited with the slider get a check. The flat tiles are the only places the clip actually stops the push.
Guess first!
r = 1.3, the crossing was better than expected (A = +1), ε = 0.2. Does this crossing still push the habit?
No. The route is already 30% more likely than when the crossing happened, past the 1.2 rope. The objective is flat there (it reads 1.2 × A no matter how far you go), so the slope is zero. Case 5.
Guess first!
r = 0.7, A = +1: a good crossing whose route has become less likely. Push or no push?
Push. Outside the ropes, but moving back toward them fixes a mistake, and PPO always allows that: here the unclipped term is the smaller one, so the min picks it and the slope is A. Case 3. The ropes only cap good news; they never hide bad news.

Reuse one batch, with and without the ropes

64 crossings are collected with a no-preference habit. Then PPO takes full-batch gradient steps on that same batch, one per epoch. Watch each route's ratio. With the ropes on, ratios mostly stop near the ropes; with them off, they keep going. "Mostly" matters: the clip removes the incentive to push further, it doesn't enforce a hard bound, so the other routes' still-active crossings can drag a clipped route past its rope (try a few new batches). That's why real PPO also watches how far each update moved, as KL divergence (step 6).

Epochs on this one batch (K)
furthest any route drifted (new ÷ old)
expected crumbs per crossing, new habit

The cliff: 60 ducklings, 15 rounds of PPO

Now real PPO training: each round, collect 12 crossings with the current habit, then run K epochs on them. Sixty ducklings with ropes, sixty without, same luck. It's on-policy, so a duckling that stops choosing route B stops seeing B's crumbs, and rarely learns it was the best route. Watch the rounds play out: the damage is done early.

Epochs per batch (K)

With lane ropes

Without

ended favouring B (the best route) · still undecided · stuck: B below 10%, it has stopped trying the best route
Guess first!
Reuse each batch 40 times with no ropes. Compared with the roped flock, what happens?
More get stuck. Twelve crossings is a small batch; sometimes B has an unlucky run in it. Squeeze that batch 40 times with no ropes and the habit swings hard away from B, and since it's on-policy, B is almost never sampled again, so the mistake is rarely corrected. That's the course's "falling off the cliff". With the ropes, one misleading batch can move the habit only so far. Pick K = 40 and run it.
Say it out loud
PPO compares the new habit with the old one as a ratio and stops giving credit once it leaves 1 ± ε, but always lets a mistake be undone. That's what makes reusing each batch many times much safer.

4 · The λ dial: how much to trust Mama

A real robot doesn't cross a pond in one go. Microduck collects a rollout of 24 steps from each simulated duck, then stops for an update. To score step 5 of those 24, you can use just Mama's one-step surprise (steady, but only as good as her guesses), or every real reward to the end of the slice (honest, but noisy), or something in between. GAE adds up Mama's surprises along the boardwalk, fading each later one by γλ.

Not explained in the course. The HF course uses GAE without ever explaining it: it appears as gae_lambda=0.98 in Unit 1's PPO settings and as an uncommented loop in Unit 8's CleanRL code. Every real PPO uses it, Microduck's included. Source: Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation (2016).

Fills the gap between Unit 6 (the one-step TD advantage) and Unit 8 (PPO)

What happens on the boardwalk
λ
γ
Score plank
Mama's one-step surprise δ on each plank · the advantage GAE gives each plank at your λ · gold curve = how much each later plank's surprise counts toward the plank you're scoring
advantage at your plank
planks until a surprise counts half
λ = 0 → · λ = 1 →

The trade: noise vs trusting Mama

200 walks along the same boardwalk, each with noisy crumbs (±0.5 per plank), scored at plank 1. Low λ leans on Mama's guesses: little noise, but her mistakes go straight in. High λ sums real rewards: honest, but the noise piles up.

Spread of the plank-1 advantage across 200 noisy walks, for each λ. The dot is your λ.
Guess first!
"A stumble at plank 15", λ = 0. What does the stumble do to the advantage at plank 5?
Nothing. At λ = 0 the advantage is just that plank's own one-step surprise, and plank 5 was an ordinary step, exactly as Mama expected: 0.00. At λ = 0.95 the stumble ten planks later reaches back at weight 0.9405¹⁰ ≈ 0.54 and drags plank 5 down to about −0.60. Which is right depends on whether the stumble was plank 5's fault, and that is exactly the bias-noise trade.
Guess first!
Microduck uses γ = 0.99 and λ = 0.95. After how many planks does a later surprise count half as much?
About 11. γλ = 0.9405, and 0.9405¹¹ ≈ 0.5. So each advantage mixes roughly the next 20 steps of real surprises, and every one of those surprises is itself measured against Mama's guess, which at γ = 0.99 stands for about 100 steps of future. Mama still supplies most of the target: why a good critic matters so much.
Say it out loud
GAE adds up Mama's one-step surprises, fading each later one by γλ. λ = 0 trusts only the next step; λ = 1 waits for the real total. Microduck's 0.95 uses about 20 steps of real rewards, then Mama's guess for everything beyond.

5 · The scorecard: RL does what you pay for

The algorithm is done. What decides whether a robot walks or does something silly is the reward function, a weighted list of terms. Here a toy robot duck can learn one of five behaviours. You write the scorecard; a policy-gradient learner then settles on whatever it pays most for. Nothing more, nothing less. That's reward hacking, and it isn't a bug.

Teaching model. Each behaviour's score on each term is a hand-written number, not physics. The learner is exact policy gradient on the five choices. Term names and the sign rules come from Microduck's AGENTS.md playbook.

Not in the course · from the reward-design rules in pollen-robotics/microduck_rl's AGENTS.md

Task
Two kinds of penalty, from the playbook. A cost (mjlab style) returns positive numbers, so it needs a negative weight. A self-negating penalty (*_penalty) already returns negative numbers, so it needs a positive weight. Get it backwards and the penalty becomes a payment. The check that always works: in the training logs, every penalty must read ≤ 0.
What the robot learns
Episode_Reward/<term> for the behaviour it chose
Guess first!
Walking task: set track_velocity (the reward for moving forward) to 0 and train. What does it learn?
Stands still. With nothing paying for progress, "upright and calm" is the best deal on the card: 49.5 against walking's 37.5. The robot has no idea what you meant.
Guess first!
Put action_rate_l2 (a cost, returns positive numbers) at +0.5 instead of −0.5. It still learns to walk. Is everything fine?
No. The behaviour looks fine, but Episode_Reward/action_rate_l2 reads +5: you're paying it to be jerky, and it will get jerkier as training goes on. That's why the playbook's check is on the logs, not the video. Try it and watch the audit turn red.
Say it out loud
RL optimizes the letter of the reward, not what I meant. Every penalty must read ≤ 0 in the logs, and I check what each lazy pose would still earn before I train.

6 · The real duck: read an actual PPO config

Microduck is an open-source ~800 g, 25 cm bipedal robot duck whose walking, standing-up and roller-skating policies are trained with PPO in simulation, then run on the real robot. Below is its actual training config. Every line is something from the pond. Click any setting.

From pollen-robotics/microduck_rl @ d424a0c · library behaviour checked in leggedrobotics/rsl_rl @ 857de61


      

    

  

  

    
What one update costs

    

      

        
Simulated ducks (envs)

        
Training rounds (iterations)

        
Fixed from the config: 24 steps per duck · 4 minibatches · 5 epochs · 50 steps per second

      

      

        

          
steps per update

          
steps per minibatch

          
gradient steps per update

          
duck-time per update

          
total duck-time, whole run

          
vs a 64-duck, 5-round smoke test

        

      

    

    

      
Guess first!

      
Microduck collects 4,096 ducks × 24 steps for every update. At 50 steps per second, how much duck experience is that?

      

      
About 33 minutes. 98,304 steps ÷ 50 per second = 1,966 s. The GPU simulates it in a moment. A full 50,000-round run is about 3 years of duck time, which is why the playbook insists on a 64-duck, 5-round smoke test first: it's 1/640,000 of the run.

    

  

  

    
Why randomize the physics?

    
The simulator's floor is never exactly the real floor. Toy version: the duck picks a balance gain and how cautious to be. Train it on one exact friction, or on a range (domain randomization), then set it down on a "real" floor.

    
Teaching model. A one-number balance task, not MuJoCo. The trade it shows is real: robustness costs some peak performance.

    

      

        
Training floors: friction 1.0 ±

        
The real floor's friction

      

      

        
        
trained on one floor (friction 1.0) · trained on your range · below zero = it falls · dashed line = the real floor

        

          
one-floor duck on the real floor

          
randomized duck on the real floor

        

      

    

    

      
Guess first!

      
The one-floor duck scores 1.0 on its own floor. Put both ducks on a floor 15% grippier (1.15). What happens?

      

      
The one-floor duck falls (−1); the randomized duck scores about 0.66. On the exact training floor the randomized duck is a bit worse (0.75 vs 1.0): it traded peak for robustness. Microduck randomizes mass, friction, joint offsets, sensor tilt and pushes for the same reason.

    

    

      
Say it out loud

      
Every knob in a real PPO config is something from the pond: γ, λ, the clip, K epochs × minibatches, 24-step slices, entropy for exploring. The rest is engineering: normalize the senses, randomize the physics, smoke-test before spending a GPU-day.

    

  







  

    

    
★ The reward review board

    
Three training runs went wrong, each a real failure from Microduck's AGENTS.md playbook, where the rules were "each learned the hard way". Read the evidence, fix the scorecard, re-train, and say why it broke. A case passes only if the robot does the right thing and the fix is the right kind of fix.

    
☆☆☆

  

  

  

  







  

    
✓ Field test

    
Eight questions. Answer each by operating the widget in the step named, not from memory. Six or more earns the check.

    

    

  

  

    
What's exact here, and what's a teaching model

    
    
      
  • Exact, real algorithms: discounting and returns; REINFORCE and the baseline (the scatter is the real spread of the estimator); the TD error; GAE, checked to equal the one-step TD error at λ = 0 and the full return minus V at λ = 1; the clipped objective and its slope in all six cases; PPO's clipped update, epochs and batch reuse; the batch arithmetic; Microduck's config values, pinned to a commit.
    • 
      
  • Teaching models: the three pond routes are made-up reward distributions. The robot behaviours in step 5 and the review board are hand-written score tables; the learner on top of them is exact policy gradient. The attempt-tax curriculum and the domain-randomization task are one- and two-number toys, not physics.
    • 
      
  • The directions are real; the duck isn't Microduck. Nothing here predicts how a real training run behaves.
    • 
    

    
Sources: the Hugging Face Deep RL Course (Units 1, 2, 4, 6, 8) · Schulman et al., GAE (2016) and PPO (2017) · pollen-robotics/microduck_rl and its AGENTS.md · leggedrobotics/rsl_rl. Further reading: Distill's Paths Perspective on Value Learning for TD vs Monte Carlo, and Lilian Weng's Reward Hacking in RL.