Showtime: Inference
By the end you'll be able to
- Trace the full path: tokens → map → layers and attention → candidates → select → repeat.
- Explain that inference runs on frozen weights — performing, never practising.
- Explain why long inputs and outputs cost more — and budget accordingly.
- Predict
- Trace
- Decide
- Field note
⚡ 30-second warm-up — From Lesson 1 — the question this whole act has been sharpening: what beats “does the AI know this?”
“Was this generated or verified?” Today you'll watch the generating happen end to end.
Follow your question through the entire machine, token by token, and watch an answer get built live.
Start with the typing illusion
When an assistant “types” its answer gradually across your screen — is that theatre, a loading animation for reassurance? Or is something really happening one piece at a time?
Commit to a prediction
The gradual arrival of the answer is…
This lesson changes one thing: you'll stop imagining a composed answer being typed out, and start seeing the whole pipeline running once per token — hundreds of full runs per reply, on dials that froze before you ever asked.
Discover the mechanism
Here is the second after you press send. Your message — plus the conversation so far and the provider's hidden instructions — is chopped into tokens (Lesson 3). Each token takes its coordinates on the meaning map (Lesson 7). The stack of layers runs, spotlights weighing every token against every other (Lessons 6 and 8). Out comes a ranked candidate list (Lesson 4); one token is selected; and that single token is the first piece of the answer.
Then the whole pipeline runs again with that token appended. And again. A 300-word answer is roughly four hundred full runs of everything you've learned — which is why text streams in gradually, why long answers cost more, and why speed varies. This performing-not-practising phase has a name: inference. The weights stay frozen throughout (Lesson 5): nothing you say tonight changes what the machine is.
So the streaming is (mostly) honest: each token appears roughly as it is computed. The machine cannot know its own ending in advance — there is no separate “thinking step” hiding behind the cursor. The loop is all there is.
two frontier caveats: some products buffer or post-process before display, so the stream you see isn't always the raw loop; and “reasoning” models first generate hidden working tokens — a private draft, also produced by this same loop, also metered and billed — before the visible answer begins (this lesson's Go deeper).A fully rehearsed theatre company. Rehearsal (training) is over — nothing tonight changes the company. Each line is performed in response to everything said so far, and once spoken, becomes part of the play the next line must follow.
- Rehearsal = training (Lesson 5). Performance = inference — tonight, for you.
- The script-so-far = your conversation, growing line by line.
- Where it breaks: actors remember tonight's show tomorrow and improvise with intent. The model retains nothing after the curtain — what assistants “remember” between chats is separate engineering (Lesson 15) — and performs Lesson 4's raffle with neither intent nor awareness.
Trace the pipeline yourself
Conceptual illustration — scripted for clarity; stage read-outs are simplified. The question is “Why is the sky blue?”. Step the machine and watch every lesson you've done run once per token.
Ready · Loop · Tokens billed:
Back to your Lesson 1 note
In Lesson 1 you explained the machine as a sentence-finisher. Now that you've seen the full pipeline: what did that analogy leave out? Most learners add two things — the machinery inside each guess (maps, layers, spotlights), and the loop feeding its own output back in. If your note already covered those, it has survived its second stress test; the third comes when search arrives in Lesson 13.
Where this bites at work
Luka's board asks why the AI bill doubled this quarter. He knows where to look now: output is metered per token, and the team's default became “write the full report” where “give me the headline figures” would do. Long in, long out — every token a full pipeline run.
The habit: ask for the length the task needs — economy as optimisation, not brevity for its own sake (Lesson 20 completes the costing frame).
Now make the call
Luka's board wants the doubled AI bill explained and fixed before next quarter — without losing the productivity gains. What does he recommend?
Try asking the assistant about its own bill — the introspection trap's final form.
Act 2 challenge: the vendor pitch
Monday. Your CEO forwards a vendor email: “Our AI has read your entire industry, learns from every conversation with your team, and its answers are never random.” — “Worth a look?” Everything from Acts 1 and 2, one decision at a time.
Apply it to your work
Run the token & cost audit on your team's three most frequent AI tasks. Set a default requested length for each — “headline figures”, “one paragraph”, “full draft” — and watch the bill follow the habit.
Prove it to yourself
Two quick questions — untimed, retryable, and they count toward ★ Mastered (completed + decision + self-check all correct). This is what defeats the feeling of "that made sense" evaporating by next week.
Explain it in your own words
Two or three sentences for a colleague. Then compare against the rubric — it checks ideas, not wording, and only you see your note.
✓ Lesson 9 stamped: Pipeline Tracer — Act 2 complete
You traced a question through everything: chopping, mapping, weighing, ranking, selecting — seventeen times over. The machine is complete, and completely sealed. Act 3 builds the scaffolding around it.
Evidence & review — how we know what this lesson claims
| Claim | Type | Basis | Review risk |
|---|---|---|---|
| Autoregressive inference runs the full forward pass per generated token | How it works | Primary research literature | Low |
| Streaming output roughly reflects computation order; some products buffer or post-process | Product behaviour | Provider documentation | High — reviewed quarterly |
| Reasoning models generate hidden intermediate tokens, also metered | Product behaviour | Provider documentation | High — reviewed quarterly |
| Inference cost scales with input and output tokens | How it works / pricing practice | Provider pricing documentation | Medium |
| This page's pipeline trace | Teaching device | Scripted illustration, labelled | Medium |
| The performance-night analogy | Useful mental model | Limits stated in the lesson | Low |
Concept review due: January 2027. Product-behaviour claims: quarterly.
Where you are in the machine
Act 2 complete — the whole engine, traced. Watch the map light up, then meet the scaffolding acts.
The machine, bottom to top: training data (2) → token pieces (3) → prediction engine (1, 4–9) → the desk (10–11) → senses (12) → live knowledge (13) → tools (14) → memory (15) → agent loops (16) → human judgement (17–21).
The trail continues: the sealed machine
You now command the complete engine — and it is completely sealed: frozen at training day, amnesiac between chats, unable to touch the world. Act 3 is the story of the scaffolding built to fix all three: the desk, the briefing, the library card, the hands, the notebook, the loop. Lesson 10 starts with the desk.