as streamed · 1280×720
Follow one camera frame through a real robot duck's vision stack: turn it, letterbox it, slide filters over it, drown in 2,100 guesses, shred the duplicates, squeeze it into 8 bits, and turn the box that survives into a head turn. Every robot number comes from its own code, and every toy is labelled as one. About 90 minutes of play.
A robot doesn't "see". It pushes one camera frame down a pipeline and gets back a number it can act on. This lab follows one frame through Microduck, a 25 cm open-source robot duck whose head camera finds other Microducks and turns the head toward them. Its head is drawn as a two-storey workshop staffed by ducklings in hard hats: every room is one step of this lab. Every robot number on these pages comes from the robot's own code, and every toy is labelled as one. Every term with a wavy red underline is tappable.
Covers HF CV Course, Unit 0–1 overview: what the course is for, and the shape of a perception pipeline · study guide 00-perception-spine.md
Put another Microduck somewhere in the room (drag it on the map, or use the sliders), then press Take one look and watch one frame travel through every room. Everything shown is computed live: the frame, the turn, the letterbox, the 2,100 candidates, the clean-up, the bearing, the angle.
The legs tick 50 times a second. The eye looks twice. Slide the looks-per-second dial and watch the board heat up; then switch the darkroom back to the way it used to work, when converting the whole frame before shrinking it ate 345 of the 407 ms a look took.
A camera doesn't hand the robot a picture. It hands it 2,764,800 numbers: a grid 1280 wide by 720 tall, lying on its side, with three numbers per cell for red, green and blue, each 0–255. The darkroom's job is to make that grid look exactly like the ones the detector was trained on: turn it upright, shrink it by one factor onto a square grey card, keep the colour order. Nothing in here is clever. Everything in here is a contract, and breaking it doesn't crash anything, which is the problem.
Covers HF CV Course, Unit 1: image as a function and an array, imaging, pre-processing (resize, pad, channel order, normalise) · study guide U1-fundamentals.md
Hover or tap anywhere on the frame (arrow keys nudge by a pixel). The loupe shows the 11×11 pixels around your finger, and the readout shows what the computer actually has there: three whole numbers, and the position they live at in the array.
The sensor streams 1280×720 on its side. The darkroom turns it a quarter turn clockwise, then fits the 720×1280 print into the model's 320×320 square. Two ways to do that: letterbox (one scale for both axes, grey where the picture runs out) or stretch (a scale per axis, no grey, ducks squashed). The model was trained on the first.
The contract has three clauses: grey 114, RGB, letterbox. Break any of them below. Nothing errors. The preview still looks like a picture. The detector still returns boxes. Only the score moves, and you'd never see it without a held-out test set.
Shrinking 720×1280 to 180×320 means every input pixel stands for a 4×4 block of the frame. Nearest-neighbour copies one pixel from the block. Bilinear blends the four around the sample point. Drag the square on the thumbnail to compare the same 40×40 patch both ways.
The detector answers in input pixels (the 320 square). The robot needs frame pixels (720×1280) to compute a bearing. Undo the mount: subtract the side pad, divide by the scale. Type the frame box for the highlighted input box.
The darkroom hands over a 320×320 print. The stencil room never looks at the whole thing at once. A duckling holds a 3×3 stencil over one spot, multiplies the nine pixels under it by the nine numbers on the stencil, adds them up, and writes the answer on a new sheet, the feature map. Then it slides over and does it again, 102,400 times. Stack a few dozen of those layers, each learning its own stencils, and they stop finding edges and start finding bills, heads and whole ducks. That stack is a CNN, and yolo11n's is what Microduck runs.
Covers HF CV Course, Unit 2: CNN intro (kernels, stride, padding, feature maps), ResNet, MobileNet, transfer learning, and the YOLO page's grids · study guide U2-cnns.md
This is the grayscale of the letterboxed input, grey-114 card and all. Pick a stencil or type your own, then hover or tap any cell of the feature map to see the nine multiplications that made it. Exact: the same cross-correlation every deep-learning library calls “convolution” (nobody flips the kernel).
Four knobs decide it. Slide them and watch the window walk across the print. The study guide's worked 5×5 (the course's Prewitt stencil, slid with no padding) is one button away; its top-left answer is 13.
core.conv2d on the numbers shown (padding cells are 0). The size rule is not written down in the course; its Keras summary (28 → 26 → 13 → 11 → 5) obeys it.A stride-2 stencil halves the print. Do it five times and 320 becomes 10. Each halving also doubles what one cell stands for, its receptive field. YOLO keeps the last three prints, 40×40, 20×20 and 10×10, and gives every cell one pigeonhole downstairs: 2,100 in all.
A standard 3×3 stencil taking 32 channels to 64 is really 64 stencils, each 3×3×32. MobileNet's depthwise-separable version splits the job: one 3×3 stencil per input channel (space), then a 1×1 stencil that mixes channels. Same shape out, a fraction of the arithmetic.
pet-detect is a ~20 KB CNN that hears the head being scratched: two 3×3 conv layers over a 40-band × 100-frame log-mel spectrogram, frequency down one axis and time along the other. Same slide-multiply-add; the print just happens to be a sound (from the study guide's reading of pet-detect/training/train.py).The stencil room's last three prints come downstairs as a wall of 2,100 pigeonholes: 1,600 small, 400 medium, 100 large. Every hole posts one sticky note per look, a candidate box and a score, whether or not it saw anything. Nobody at the desk throws a note away. That is YOLO's bargain: one pass, a fixed number of guesses, and a clean-up afterwards. On Microduck the clean-up is decode: a gate (the threshold), a sort, the shredder clerk (NMS), and a trip back out of the letterbox. Then the inspector grades the whole desk with one number, mAP50.
Covers HF CV Course, Unit 6: object detection, IoU, mAP, a segmentation aside · and Unit 2's YOLO page (grids, NMS) · study guide U6-detection-and-yolo.md
One frame: a duck straight ahead. The heat is every hole's score on the floor you pick; the gold outlines are the notes that clear the gate. Tap any hole to read its note, pulled straight out of the raw planar tensor by index.
Same frame, same notes. The clerk sorts the ones past the gate by score, keeps the first, and shreds every later note whose IoU with a kept note reaches the limit. Step through it, or run it, then raise the limit and watch duplicates survive. Exact: core.decode is the robot's decode, including its stable sort.
decode undoes the darkroom's letterbox so the box lands on the upright 720×1280 frame. (The real code un-letterboxes every note before the sort. Shifting and scaling all boxes alike leaves every IoU unchanged, so the order doesn't change the answer.) The daemon keeps it upright “because that is the space the boxes are in” (mediad/src/detect.rs:260).Drag your box over the true duck box; drag the corner square to resize. Brick = the overlap, both tints together = the union. Every grid unit is one pixel, so the arithmetic is exact and the challenges have exact answers.
The inspector grades a whole recording the detector never trained on, a held-out session: 40 frames of the toy room with real ducks, rubber ducks and wall prints, scored by the same precision–recall arithmetic as any detection benchmark and summed up as AP50. Slide the gate and watch TP · FP · FN move; switch the look-alikes off to see what hard negatives cost.
The head's output is [1, 5, 2100]: five long rows, not 2,100 short records. Candidate i's numbers sit at raw[k·2100 + i]. Read it as five numbers per box and all five values come from one row, five neighbouring holes' cx (or cy, or w…). Four in five of the “scores” are really pixel coordinates or box sizes, far over any threshold, and many of the boxes are giants. It doesn't crash. It looks like a weak model.
Up at the box desk every sticky note was written with a float32 pen: any value, seven digits. The NPU is built for a cheaper pen: 8-bit integers, only 256 of them. So before the model boards the robot it is quantised: every weight and every output is measured with a tiny ruler that has 256 notches, and the gap between notches is set by the biggest number that ruler has to reach. Choose the ruler well and you lose at most half a notch. Share one ruler between 320-pixel box coordinates and 0–1 scores, and the scores vanish. That is Microduck's "about 1.3" bug, and it is written into the real config.
Covers HF CV Course, Unit 9: model optimisation (quantisation, pruning, distillation), deployment (ONNX, edge), tools · study guide U9-model-optimization.md
Pick a float range, pick how many bits, and the ruler is fixed: the scale is the gap between notches, the zero point is the notch that stands for 0.0. Drop a value on it and read what comes back. Inside the range the error is never more than half a step. Outside, the value is clipped to the end of the ruler.
scale, zero_point off the model. The course names quantisation without ever showing it; the study guide's worked example fills that gap, and this ruler reproduces it.A model file is roughly parameters × bytes per parameter. Four bytes for float32, two for float16, one for INT8: the course's "about 4 times". Microduck ships both files side by side, so we can check.
The real config says it plainly: · . Here is the mechanism, on the teaching detector's raw [5, 2100] output for a duck scene. The box rows need a ruler that reaches ~320 px. One ruler for the whole tensor means one notch ≈ 1.26, and a 0–1 score can only land on notch 0 or notch 1.
Forty frames the detector never trained on (35 ducks, plus rubber ducks and wall prints as look-alikes). Same frames, three ways of storing the head's output.
threshold = 0.35 is a real probability cut when the float .onnx runs on the CPU (the file size says it is float; our inference). On the NPU the same number is an on/off switch.A look costs about 60 ms, so speed alone would allow ~16 looks a second. The robot takes two. · . Slide the rate and watch the board, the clock, and the 50 Hz walking loop that shares the CPU.
rknn-toolkit2 to convert, rknpu2 to run) appears nowhere in Unit 9. The nearest analogue is the Edge TPU notebook: full-integer post-training quantisation with a representative dataset, then a vendor compiler for one chip. The per-tensor-vs-per-channel choice that decides the 1.3 bug isn't in the course either; the study guide covers it.The box desk hands the neck crank a bearing: where the box sits across the frame, −1 to +1. That is a position on paper, not an angle in the room. To turn it into "9.6° to the right" you need the camera's intrinsics, the numbers that say how the pinhole maps the world onto the sensor. Microduck's are still placeholders, and this mount has a trap: the camera is turned on its side, so the sensor's famous 62° runs up and down, and the left-right view is at most about 37°.
Covers HF CV Course, Unit 8 (part 1): camera models, intrinsics, linear algebra for 3D · study guide U8a-camera-geometry.md
Drag the other duck around the room. The camera frame on the right shows exactly where the pinhole model puts it, with its box. The readouts turn that box into an angle four ways: the truth, the correct intrinsics, a placeholder that puts "62° across the width" (our guess at what a placeholder would do), and the lazy linear guess.
The IMX219 is a 4:3 chip, 3280×2464. Every 16:9 video mode has to throw rows away, and some throw columns away too. Then the mount turns the picture a quarter turn. Which mode the robot runs decides the true left-right field of view, and the repo doesn't pin it down yet.
mediad runs is a question for the team.The design doc's range estimate: · . Three ways to get that number, all in one box, with a distance slider.
The projection only ever uses the ratios X ÷ Z and Y ÷ Z. Scale a point away from the camera along its ray and it lands on exactly the same pixel. That is why a single camera can say "which direction" and never "how far", and why step 6 brings in the tape measure.
kinematics crate does it for the head: it stores each pose as a position plus a quaternion, the same information (our reading of kinematics/src/head.rs); the ToF points reach the trunk that way too (). If you built the Arm Playground, the extrinsics here are a recap.The map room turned a box into a bearing. It cannot turn a box into a distance: a pinhole keeps only the ray, and every point along a ray lands on the same pixel. This step is about getting distance back. People do it with two eyes (stereo). Networks try to guess it from one image (monocular depth), and usually get the shape but not the metres. Microduck's plan sidesteps both: the tape measure with 64 beams measures distance directly, and the radio mast says which duck it is, because the camera cannot tell two identical Microducks apart. The last widget makes you do the fusion yourself.
Covers HF CV Course, Unit 8: stereo vision, disparity, monocular depth (relative vs metric) · Unit 10 (one line): synthetic data with BlenderProc · study guide U8b-depth-and-stereo.md
A 10 cm rubber duck and a 25 cm Microduck. Slide the rubber duck toward the camera and watch its box grow. At one distance it is exactly as tall in the picture as the robot duck at 3 m. The known-size cue Z = f·H ÷ h can turn that height back into metres, but only if you already know which object it is.
Two cameras a baseline B apart see the same point at slightly different pixels. That shift is the disparity d, and depth is Z = f·B ÷ d. The catch is in the derivative: one pixel of disparity error moves the depth by about Z² ÷ (f·B), so the error grows with the square of the distance. The course's numbers are the OAK-D Lite's: f = 452.9 px, B = 7.5 cm.
Microduck's ToF sensor ranges an 8 × 8 grid of zones over a 45° × 45° square: 64 slant distances, once per scan. The robot's kinematics/src/tof.rs turns each zone into a point and then throws two kinds away: beams that reached the floor (a head looking down sees floor at every range), and returns under about 10 cm, where cover-glass crosstalk makes phantoms. Drag the objects around the plan view. The grid on the right is what the sensor would publish, filtered as the code filters it.
Zone::Hit.range carries. Click a row to draw it.A network can learn depth from one image, but most are trained with a loss that first normalises every depth map (subtract the median, divide by the mean absolute deviation). That makes the training data from a hundred cameras agree, and it makes the model blind to scale and shift. Drag the two sliders: whatever scale and shift you apply, the loss stays zero.
Two identical Microducks are in view. The box desk found both and gave each a bearing; it cannot say which is which. The tape measure gives a distance along each bearing. Jo's radio beacon says Jo is present, and how strong the signal is. Put the three together, click the box you think is Jo, set the neck crank, and turn the head.
Three things that go wrong in Microduck's vision stack, each one a warning written into its own source. For each: read the evidence, turn the knobs, re-run the real pipeline (the same detect → quantise → decode → AP functions as the earlier steps, on the 40-frame held-out session), and name the cause. A case closes only when the behaviour is fixed and the fix is the right kind. Tempting wrong fixes are on the board too; they fail for a stated reason.
Eight questions. Answer each by operating the widget in the step named, not from memory: set its controls to the values in the question and read the number off the page. Numeric answers have a tolerance. Six or more earns the check.
lib.rs; IoU, precision, recall and AP (all-point and 101-point); affine INT8 quantisation (scale, zero point, per-tensor vs per-channel); convolution, its output size and parameter counts; the pinhole camera, field of view and stereo maths; the ToF beam table and its floor and too-close filters; and every robot fact, pinned to pollen-robotics/microduck @ 590b986 with its source line.[1, 5, 2100] layout and the three grids are real, the numbers are not yolo11n's, and its 0.979 is a toy AP); the thermal curve (only the 95 °C / 408 MHz flat-out anchor is real); the camera and sensor sitting 0.20 m above the floor; which sensor mode the camera runs (we assume the widest full-width crop; the 1080p crop is hypothetical) and what the placeholder intrinsics are (we assume 62° across the width); the duck-head stereo rig; and the ToF + BLE fusion toy, including "strong signal = near".duck-detect/src/lib.rs, deploy/robotd.toml, docs/project/npu-bringup.md, docs/ideas/autonomous_behavior.md, kinematics/src/tof.rs.