The Dice Inside the Machine
Watch the machine hesitate — and learn the single setting that makes AI creative or careful.
By the end you'll be able to
- Explain that for each next token the model produces a ranked list of candidates with likelihoods.
- Show how sampling settings trade consistency against variety.
- Explain why the same prompt can give different answers on different runs — by design.
- Predict
- Explore
- Decide
- Field note
⚡ 30-second warm-up — From Lesson 1: what did asking the assistant “Are you sure?” actually produce?
Another generated continuation — not independent confirmation. Verification lives outside the model.
Start with a small mystery
Ask the same AI the same question twice, and you may get two different answers. No settings changed, no new information. Is it changing its mind?
Commit to a prediction
Same model, same prompt, different answers. What's the most likely explanation?
This lesson changes one thing: you'll stop reading different answers as unreliability and start seeing a weighted selection you can tune — consistent for compliance work, varied for ideation, and never a verdict on truth.
The one idea in this lesson
At each step, the model doesn't pick one next token. It produces a score for every token in its catalogue — a ranked list from very likely to absurd. Then a separate rule chooses from the list.
Always take the top candidate, and output is consistent but can be flat and repetitive. Sample from a wider spread — controlled by a setting usually called temperature — and output becomes varied: sometimes better, sometimes unhinged. Identical prompts giving different answers is this rule working as designed.
One more thing to notice, because it becomes Lesson 17: when the model's evidence is thin, the spread is wide and flat — yet a token still gets selected. Unless another behaviour intervenes — a refusal, a clarifying question, a tool call — the model is optimised to keep generating, even when the available context cannot establish the answer.
real products layer several sampling rules on top of temperature, many assistants don't expose the dial at all, and frontier "reasoning" models run this same raffle over their hidden working steps too. The full answer-building loop runs this raffle hundreds of times per reply (see the map below); what happens when a flat raffle meets a factual question is Lesson 17.A picture to hold on to
A raffle drum, where better-fitting continuations get more tickets. Temperature reshapes the allocation: cold, and the favourite holds nearly every ticket; hot, and tickets spread widely across the field.
- Tickets = probability assigned to each candidate token.
- The draw = choosing one token.
- Refilling the drum after every draw = the loop from Lesson 1, running again for the next piece.
- The ticket printer = the trained model. (How it learned to print tickets so well is Act 2's story.)
The temperature laboratory
Conceptual illustration — not probabilities from an actual model. Two prompts, one dial. Move the temperature, watch the tickets redistribute, then draw — repeatedly. Predict before each draw: will you get the same word as last time? (The draws use real randomness because experiencing variability is the point. Nothing is sent anywhere.)
Where this bites at work
Maya designs client-facing AI assistants at a financial advice firm. For compliance-checked client explanations, she locks the variability down: the same question must produce the same approved shape of answer every time. For internal campaign brainstorming, she opens it right up — surprise is the product.
Same model, opposite settings, both correct. The dial isn't good or bad; it's task-fit. And run-to-run variation is never evidence that the machine "changed its mind" — nothing in there has a mind to change.
Where the picture breaks
A raffle is pure chance. Here, the ticket allocation encodes everything the model learned — the favourite is usually excellent, which no fair raffle can claim. The randomness is only in the draw, not the weighting.
Also: "drawing" is one decoding method among several — real systems select or sample from the distribution depending on their decoding settings (greedy top-pick, temperature sampling, top-p and more). Many tools don't expose any dial, and even "maximum consistency" settings aren't always perfectly identical run to run in deployed products. The concept is durable; the knobs vary by product.
Now make the call
In testing, Maya's client-facing assistant answered the same fee question three times with three differently-worded replies. Compliance wants approved, consistent wording every time — this is a financial advice firm. What does she do?
Try "ask the assistant why" — it's the third appearance of the introspection trap, and the last time it should ever catch you.
Act 1 challenge: Friday, 4 pm
Board papers are due at 5. Your assistant's draft summary includes: “The Harwood Trust grant (applications close 30 September) totals $48,500.” Everything you've learned in Act 1, one decision at a time.
Apply it to your work
Sort this week's AI tasks into two piles: consistency tasks (board papers, compliance wording — turn variability down and supply an approved template) and ideation tasks (campaign angles, naming — open it up). Record both piles in your capstone checklist starter.
Your habit from this lesson
Match variability to the task: consistency tasks cold, ideation hot — and check whether your AI tool exposes the setting. Above all, stop reading run-to-run variation as lying, learning or mood. It's a raffle you can tune.
Go deeper (optional): the maths under the dial
The model's raw scores are turned into probabilities by a function called softmax; temperature divides the scores before that conversion, which is why low values exaggerate gaps between candidates and high values flatten them. Products often combine temperature with top-p sampling — ignore the long tail of terrible candidates and draw only from the smallest set covering, say, 90% of the probability.
Why isn't "temperature 0" always perfectly identical in real products? Deployed systems batch work across many users on parallel hardware, and tiny arithmetic-ordering differences can occasionally flip a near-tie. The concept — deterministic favourite-picking — holds; the engineering adds noise at the margins.
Deeper challenge: in the laboratory above, find the temperature at which the creative prompt's five-draw run first produces four different words. Then explain, in raffle terms, why the factual prompt won't do that at any temperature.
Back to your Lesson 1 note
In Lesson 1 you wrote your own explanation of what produces an AI answer. Now that you've seen the raffle: would you change anything? Most learners add something about the spread — that answers aren't just "the most likely continuation" but a selection from a weighted field, tunable between consistent and varied.
Prove it to yourself
Two quick questions — untimed, retryable, and they count toward ★ Mastered (completed + decision + self-check all correct). This is what defeats the feeling of "that made sense" evaporating by next week.
Your field note
One or two sentences: why can the same question get different answers, and when would you want each behaviour?
✓ Lesson 4 stamped: Raffle Master — Act 1 complete
You tuned the dial, predicted the draws, and separated variability from unreliability. With that, you've finished Act 1: you now know what AI is actually doing. Act 2 — how it learned to do it — starts with Lesson 5.
Evidence & review — how we know what this lesson claims
| Claim | Type | Basis | Review risk |
|---|---|---|---|
| Models produce a probability distribution over candidate tokens at each step | How it works | Primary research literature | Low |
| Decoding settings (temperature, top-p, greedy) reshape or replace sampling | How it works | Primary research and framework documentation | Low |
| Products layer sampling rules and often hide the dial | Product behaviour | Provider documentation | High — reviewed quarterly |
| "Temperature 0" is not always perfectly deterministic in deployed systems | How it works (empirical nuance) | Engineering analyses | Medium |
| The laboratory's distributions | Teaching device | Simulation, labelled; draws use real randomness deliberately | Medium |
| The raffle-drum analogy | Useful mental model | Limits stated in the lesson | Low |
Concept review due: January 2027. Product-behaviour claims: quarterly.
Where you are in the machine
You've reached the heart of the engine — and with your field note stamped, Act 1's layers light up. Select any layer for its role, lessons and your live progress.
The machine, bottom to top: training data (2) → token pieces (3) → prediction engine (1, this lesson, 5–9) → the desk (10–11) → senses (12) → live knowledge (13) → tools (14) → memory (15) → agent loops (16) → human judgement (17–21).
Next: how did it get this good?
You now know what the machine does: chop, weigh, raffle, repeat. Act 2 asks the deeper question — how did it learn to allocate those tickets so well? Nobody programmed the rules in. The story of how they got there is called training. Lesson 5: Ten Thousand Nudges.