Inference Kitchena hands-on lab

How an AI model actually gets served

Every chatbot reply is cooked in a GPU kitchen. Drag the sliders, flip the switches, and break things on purpose. The field guide and Kiely's Inference Engineering in about 90 minutes of play, with no reading required.

0 · What happens when you hit Enter

A GPU is a kitchen. Before anything else, meet the cast. Every term with a wavy red underline is tappable: hover or tap it for the plain-English meaning and its kitchen equivalent.

GPU = the kitchen
where the cooking happens
HBM = counter space
fast memory right next to the stove: small, precious
Weights = recipe books
must stay on the counter the whole time
KV cache = order tickets
one per diner, growing as the conversation goes
Inference engine = head chef
decides who gets cooked for next
Batch = a table served together
one trip to the books, many plates

Covers Field guide §0–§1 · Kiely ch.1–2 · the same ground as your "How a Call Runs" one-pager, but moving

Send one request and watch it cook

waiting…
Your prompt, as tokens (prefill reads them all at once)
The answer (decode writes one token per trip to the recipe books)
Stove busy (compute)0%
Hauling recipe books (memory)0%
0
GB of recipe book read so far
0
tokens on this diner's ticket
0 ms
kitchen time

The one rule of decode: speed ≈ bandwidth ÷ model size

Model
GPU
recipe books (FP8)
hauling speed (bandwidth)
max words/sec for one diner
Guess first!
Your prompt is 500 tokens and the answer is 500 tokens. Which half takes longer?
Writing takes far longer. The 500 prompt tokens go through in one parallel pass. The 500 answer tokens need 500 separate trips to re-read every weight, because each word depends on the one before it. That's why most closed-model APIs charge several times more for output tokens than input tokens (typically 3–8×; many open-model hosts charge the same for both).
Say it out loud
Prefill reads my whole prompt in one parallel burst. Decode writes one token at a time, re-reading every weight each time. So one user's speed is bandwidth ÷ model size, and output tokens cost more.

1 · The counter: memory decides the hardware

The recipe books (weights) must fit on the counter (HBM) before a single diner is seated. Whatever space is left holds order tickets (KV cache), one per diner, and each grows with the conversation's length. Drag the diners up until the counter overflows.

Covers Field guide §9(a)(b), §4.1 · Kiely ch.3, ch.5 · GPU specs from your Hyperscale Ledger

GPU
How many GPUs
Model
Recipe-book precision (quantization)
Ticket precision (KV cache)
Conversation length (context)
Diners seated
recipe books (weights)order tickets (KV cache)overflow = out of memory
weights
one ticket at this length
counter size
diners that fit
Guess first!
A 70B model at FP8 on one H200 seats 52 diners at 8K context. Double the context to 16K. How many fit now?
26. Each ticket doubles in size, so half as many fit. Context length is a cost multiplier, and that's why long-context tiers cost more. Set it up above and check.
Guess first!
DeepSeek-V3 has 671B parameters but only 37B are "active" per token. At FP8, does it fit on one B200 (180 GB)?
No. Every expert has to sit on the counter, even the ones a given token doesn't use: 671 GB. MoE saves compute, not memory. Pick DeepSeek and 1×B200 above to watch it overflow, then try 8×B200. The small tickets you'll see there come from DeepSeek's compressed attention (MLA), a separate trick.
Say it out loud
Memory decides the hardware. The weights must sit in GPU memory first, and every user's KV cache grows with context, so context length multiplies cost. MoE saves compute, not memory.

2 · The batch dial: one trip, many plates

Each decode step hauls every recipe book across the kitchen once. Serve one diner per trip and the stove sits idle. Serve a whole table on the same trip and each extra plate is nearly free. Turn the dial and watch your speed against the kitchen's total.

Covers Field guide §0, §3.2 · Kiely ch.2 (arithmetic intensity), ch.5 · roofline cross-checked against DeepMind's How to Scale Your Model

GPU
GPUs
Model
Precision
Average context
Batch (diners per trip)
GPU rent
your speed (tok/s)
kitchen total (tok/s)
$ per million tokens, at full load
the bottleneck
Blue = one diner's speed · green = the kitchen's total · dashed gold = where decode turns compute-bound · red zone = the tickets no longer fit on the counter
Guess first!
70B at FP8 on 2×H100. Go from 1 diner per trip to 64. What happens to your speed?
It drops only by about a third (66 → 42 tok/s at 4K context), nowhere near 1/64th, because decode is memory-bound. The expensive part is hauling 70 GB of books, and that happens once per trip however many plates ride along. The drop you do see is the tickets: each diner's 4K-token KV cache has to be read too (43 GB at 64 diners). That's why long contexts make batching less of a free lunch. Meanwhile the kitchen's total goes up ~40×, which is why per-token prices collapsed. Drop the context to 1K and your speed falls only ~13%.

The opposite trick: speculative decoding

A fast sous-chef (a small draft model) guesses the next k words. The head chef checks all of them in one trip and keeps the ones it agrees with, up to the first miss, plus one of its own. One diner gets several tokens per trip. That's different from batching, which is many diners per trip.

How often the guess is right (α)
Words guessed per trip (k)
Sous-chef cost vs head chef
tokens per trip (expected)
speedup
Formula from Leviathan et al. (2023), the paper behind the technique: expected tokens per trip = (1 − αk+1) ÷ (1 − α), then divide by the sous-chef's extra cost. It works best at small batches. When the kitchen is already compute-bound, checking extra guesses isn't free.
Guess first!
Guesses are right only half the time (α = 0.5). Is guessing 8 words per trip better than guessing 2?
2 wins (1.59× vs 1.43×). With a 50% hit rate the chain almost always breaks early, so long guesses mostly waste the sous-chef's time. Engines tune k to the acceptance rate they actually see, which is why benchmark content matters (Kiely ch.5).
Say it out loud
Batching shares one read of the weights across many users: total throughput soars while each user slows only a little, from reading their own KV cache, until decode turns compute-bound around a few hundred. Speculative decoding is the opposite trick: several tokens per read for one user.

3 · The head chef: the scheduler sandbox

Now run a real dinner service. Requests arrive all minute: mostly short, a few huge. The inference engine is the head chef deciding, every few milliseconds, who is on the counter and who gets the next trip. Flip one switch at a time and watch who waits.

Covers Field guide §1–§3, §5 · Kiely ch.4–5 · the core idea: an inference engine is a scheduler for GPU memory

Diners arriving
How the chef seats people
Chunk size
TTFT median / p99
ITL median / p99
throughput tok/s
goodput · met SLO
First 20 seconds of service, one row per diner: waiting · prefill · decoding · red outline = a giant order
Time between words for everyone, across the minute. Dashed line = the 100 ms "feels smooth" SLO. Spikes are stalls.
Guess first!
Switch on giant orders (2% of requests are 60K-token prompts). What happens to everyone else?
Everyone freezes. Without chunking, a 60K-token prefill runs inside one scheduler step, about 2.4 seconds during which nobody gets a word. Watch the spikes in the lower chart, then try chunked prefill (it spreads the giant out; watch the chunk-size tradeoff) and then disaggregation (the giant gets its own kitchen).
Guess first!
Push arrivals from 6 to 15 per second. Throughput goes up. What happens to goodput?
It collapses. The kitchen serves more tokens, but almost nobody gets them fast enough to meet the SLO. That's why serving research (DistServe) and the field guide judge engines on goodput at your SLO, not raw throughput: a benchmark that only reports tokens/sec can look great while every user is miserable.

Which engine would the head chef be?

Say it out loud
An inference engine is a scheduler for GPU memory. Continuous batching and paged KV keep the counter full; chunked prefill or disaggregation stop giant prompts from freezing everyone; prefix caching skips repeated prompts. Judge it on goodput at your SLO, not raw throughput.

4 · Big kitchens: splitting a model across GPUs

When the recipe books don't fit on one counter, you split them across several kitchens. There are three ways to cut them, and each costs a different amount of shouting between kitchens. Inside one building the hallway is fast (NVLink). Between buildings it's much slower (InfiniBand).

Covers Field guide §4 · Kiely ch.5 (parallelism) · cross-check: DeepMind scaling book, "Sharding" and "Inference" chapters

Model
GPU
GPUs
How to cut it
Diners per step
books per GPU
shouting per step
one diner's tok/s
kitchen tok/s

Why MoE changes everything: how many experts wake up?

Model
Tokens in the step
of experts read this step
weights read (FP8)
Share of experts touched = 1 − (1 − k/E)tokens. On a busy server, "37B active" becomes almost everything active.
Guess first!
DeepSeek-V3 sends each token to 8 of its 256 experts. With 64 users decoding together, roughly what share of experts gets read each step?
About 87%. Each token picks different experts, and 64 tokens × 8 picks cover almost the whole kitchen. That's the book's key MoE point: the efficiency on the model card is a single-request property. On a busy server you need expert parallelism spread over many GPUs to keep each GPU's slice small.
Say it out loud
Tensor parallelism splits each layer and is fast inside one machine but chatty across machines. Pipeline splits layers into stages and doesn't speed up a single user. Expert parallelism splits an MoE's experts and scales across machines. That's why the smallest real deployment of a giant MoE is a whole node.

5 · Run it as a business

Your AI Stack atlas has the personal version of this (the Token Meter: should you pay per token or buy a Mac?). This is the company version: should a company rent its own GPU fleet, and what does it take to run one?

Covers Field guide §6, §8, §9(c) · Kiely ch.1, ch.7 · Darden GBUS 8496 Class 10 (Tryon, "Own the Weights")

a · Fleet break-even: self-hosting is a utilization bet

GPU rent (per GPU-hour)
GPUs
Kitchen throughput when busy
API price you'd pay instead
Your traffic shape over a day
Customers pooled
your utilization
your $/M tokens
break-even utilization
verdict
Gold curve = your cost per million tokens at each utilization · blue line = the API · where they cross = break-even · brown dot = you
Your demand across 24 hours. You pay for the peak all day long.
Guess first!
The field-guide kitchen (2×H100, $2.50/hr, 2,500 tok/s) serves business-hours traffic. Is it cheaper than a $0.90/M API?
No. $0.56/M is only true at 100% utilization. Business-hours traffic keeps you at about 45%, which makes it ~$1.23/M. Break-even is 62%. Now switch on pooling: a platform with many customers across time zones flattens the curve. That's the Baseten/Together business model in one switch.

b · Replay Tryon (Class 10): your own case, moving

Tryon labels financial sentences. Owning the tuned 1.7B model costs $19.23 per million labels plus the people to run it. Renting the API costs $890 (frontier), $356 (standard) or $35.60 (budget) per million. Your class answer was "owning wins only against the tier nobody retested." Drag the volume and see where that flips.

Labels per day
API tier Tryon compares against
People to run it
API per year
owned per year
break-even labels/day

c · The failure clock

Meta's Llama 3 run saw 419 unexpected interruptions in 54 days on 16,000 GPUs: about one failure per 50,000 GPU-hours (Kiely ch.7).

Fleet size
a failure every
failures per week
So production means planning for failure: several regions serving at once (active-active) or a hot standby (active-passive), plus canary deploys. Blue-green needs a full duplicate fleet, and nobody can afford that at 100 GPUs.

d · The cold-start stack

How long until a new replica serves its first token? Shrink each bar.

time to first token for a new replica

e · Who sells what: route the request

NeocloudCoreWeave · Lambda · Nebius · Nscale
Sells GPU-hours. You bring the engine.
Inference platformBaseten · Together · Fireworks · Modal
Rents GPUs from neoclouds; sells tuned serving of open models.
RouterOpenRouter
Owns no GPUs. Sells one API and one bill across everyone, labs included.
Lab / cloud partnerOpenAI · Anthropic · Google · Bedrock · Vertex · Foundry
The only place closed models are served.
Pick what the buyer needs.
Say it out loud
Self-hosting is a utilization bet, not a price bet: you pay for the peak all day, so you only beat the API above break-even utilization, and platforms exist because pooling many customers keeps GPUs busy. In production, plan for GPU failure and cold starts.

6 · The other kitchens: same recipe books, different buildings

Steps 0–5 all happen in one kind of kitchen: a datacenter GPU with its HBM counter. The same recipe books also get cooked in a food truck (the NPU in your laptop or phone), on a prep line where every book is already open on the counter (the LPU), and in a robot's onboard computer. Before any of that, they get written in a banquet hall the size of a warehouse. Three questions sort every one of them: where do the recipe books sit, how fast can they be hauled, and how much power does the kitchen draw?

Covers Chip guide §4, §6, §6a, §13 · Physical AI guide §2.4 · Llama 3 paper §3.3

a · Which counter? Sort the kitchens

For each kitchen, pick where its recipe books live. SRAM is on the chip itself: the fastest memory there is, but it holds only megabytes. HBM sits in the package right beside the chip. LPDDR is low-power memory on the board, shared by everything on the chip.

b · Same recipe, different kitchen

Kitchen
Model
Recipe-book precision (quantization)
where the books sit
recipe books
kitchens to hold them
one diner's speed
Guess first!
An 8B model at 4-bit, running on a laptop's NPU. The chip is rated at 80 TOPS, but it reads the recipe books from shared memory at about 0.135 TB/s. How fast can it write for you?
About 24. The 80 TOPS barely matters: every word means hauling all 4 GB of recipe books once, so speed ≈ 0.135 TB/s × 70% ÷ 4 GB. That's plenty for a phone-sized assistant. A 70B model doesn't even fit on a 32 GB laptop, even at 4-bit.

c · The banquet hall: where the recipe books get written

Training is one enormous dinner service in lockstep. Every table cooks its share of the course, then all of them compare notes (the all-reduce) before anyone starts the next one. So the hall runs at the speed of its slowest table (a straggler), holds far more on the counter than serving ever does, and has to be one building. The work is counted in FLOPs; how much of the stoves' time is real cooking is the MFU.

Model
Training tokens
H100s in the hall
MFU (real cooking time)
Chance one table is slow, per step
FLOPs to write the books (6 × params × tokens)
days on the calendar
days with slow tables
on the counter to train (16 bytes/param)
to serve it at FP8
something breaks every
Red curve = chance that a step waits on at least one slow table, by hall size · dot = your hall. GPU power alone: .
Guess first!
Serving Llama 3 405B at FP8 means about 405 GB of recipe books: 6 H100s. How many H100s does it take just to hold its training state?
About 81. Serving keeps one copy of the weights at 1–2 bytes each. Training keeps five things per parameter: the weights (2 bytes), the gradients (2), a full-precision master copy (4), and two running averages for the optimizer (4 + 4). That's 16 bytes × 405 billion = 6.5 TB, or 81 H100s, before a single activation. It's why training is a different machine, and why a LoRA adapter, which trains about 1% of the weights, fits on one small GPU.

d · Three kinds of kitchen, one table

Banquet hall · trainingColossus · Stargate · Rainier
Books: 16 bytes a parameter, spread over thousands of GPUs. Bound by the math and the slowest table. One building, hundreds of megawatts. A handful of labs buy it.
Restaurant · servingsteps 0–5 · H100 · TPU · LPU
Books: 1–2 bytes a parameter in HBM or SRAM. Bound by hauling the books for every word. Goes wherever the diners and cheap power are. Everyone who serves a model buys it.
Food truck · edgelaptop NPU · phone · Jetson Thor
Books: small models at 4-bit in shared LPDDR. Bound by watts and memory. Runs where the data is, with no network. Trains nothing.
Say it out loud
Every kitchen answers three questions: where the recipe books sit, how fast they can be hauled, and how much power it draws. HBM kitchens are the restaurant. SRAM kitchens trade capacity for speed and hit the wires instead. LPDDR kitchens (laptops, Macs, robot computers) hold a lot, haul slowly and sip power. The banquet hall that writes the books is another machine entirely: 16 bytes a parameter, every table waiting for the slowest, and something breaking every few hours.

★ · You're the inference lead

Three clients, three briefs. Pick the hardware, the model, the precision and the engine features, then submit. The plan is scored on whether it fits, meets the SLO, and stays in budget, using the same math as steps 1–5. A plan is some number of replicas, identical copies of the model, each split over 1–8 GPUs inside one machine, with users shared evenly across them. Rents: A100 $1.60 · H100 $2.50 · H200 $3.50 · B200 $6.00 per GPU-hour.

GPU
GPUs per replica (one machine)
Replicas (copies of the model)
Model
Precision
☆☆☆☆☆

Configure, then submit.

✓ · Field test

Eight questions. The answers aren't on this page; you get them by operating the widgets in steps 1–5. Graded instantly, with rounding tolerance.

What's exact here and what's a model

Exact: the memory arithmetic (weights, KV cache per token, users that fit), the MoE expert-coverage formula (assuming uniform routing; real routers are skewed), the speculative-decoding formula, the break-even math, and your Class 10 numbers. Teaching models: decode speed assumes 70% of peak bandwidth and compute; the scheduler sandbox uses round-number costs (prefill ≈ 25K tok/s; a decode step ≈ 12 ms plus a little per diner); the parallelism panel approximates communication with ring all-reduce and all-to-all volumes. The directions are the real ones; the exact milliseconds are not a benchmark. The capstone's GPU rents ($1.60–$6.00 an hour) are illustrative round numbers in the range of 2026 on-demand prices, not quotes. GPU specs come from the audited Hyperscale Ledger. GB means 10⁹ bytes throughout.