The Dice Inside the Machine
Watch the machine hesitate, and learn the single setting that makes AI creative or careful.
By the end you'll be able to
- Explain that for each next token the model produces a ranked list of candidates with likelihoods.
- Show how sampling settings trade consistency against variety.
- Explain why the same prompt can give different answers on different runs, by design.
- Predict
- Explore
- Decide
- Field note
⚡ 30-second warm-up. From Lesson 1: what did asking the assistant “Are you sure?” actually produce?
Another generated continuation, not independent confirmation. Verification lives outside the model.
Start with a small mystery
Ask the same AI the same question twice, and you may get two different answers. No settings changed, no new information. Is it changing its mind?
Commit to a prediction
Same model, same prompt, different answers. What's the most likely explanation?
This lesson changes one thing: you'll stop reading different answers as unreliability. Instead, you'll see a weighted selection you can tune: consistent for compliance work, varied for ideation, and never a verdict on truth.
The one idea in this lesson
At each step, the model doesn't pick one next token. It produces a score for every token in its catalogue, a ranked list from very likely to absurd. Then a separate rule chooses from the list.
Always take the top candidate, and output is consistent, but it can be flat and repetitive. Sample from a wider spread instead, and output becomes varied: sometimes better, sometimes unhinged. A setting usually called temperature controls how wide that spread is. Identical prompts giving different answers is this rule working exactly as designed.
One more thing to notice, because it becomes Lesson 17: when the model's evidence is thin, the spread is wide and flat. Even so, a token still gets selected. The model is optimised to keep generating, even when the available context cannot establish the answer, unless another behaviour intervenes, such as a refusal, a clarifying question or a tool call.
real products layer several sampling rules on top of temperature. Many assistants don't expose the dial at all, and frontier "reasoning" models run this same raffle over their hidden working steps too. The full answer-building loop runs this raffle hundreds of times per reply (see the map below). What happens when a flat raffle meets a factual question is Lesson 17.A picture to hold on to
Picture a raffle drum. Every possible next word gets some tickets, and better-fitting words get more of them. Temperature controls how those tickets are spread. Turn it down, and the favourite word hoards nearly every ticket, no other word gets a look in. Turn it up, and the tickets spread out across many words, so several of them have a real shot.
- Tickets = probability assigned to each candidate token.
- The draw = choosing one token.
- Refilling the drum after every draw = the loop from Lesson 1, running again for the next piece.
- The ticket printer = the trained model. (How it learned to print tickets so well is Act 2's story.)
The temperature laboratory
Conceptual illustration, not probabilities from an actual model. Two prompts, one dial. Move the temperature, watch the tickets redistribute, then draw repeatedly. Predict before each draw: will you get the same word as last time? (The draws use real randomness because experiencing variability is the point. Nothing is sent anywhere.)
Where this bites at work
Maya designs client-facing AI assistants at a financial advice firm. For compliance-checked client explanations, she locks the variability down: the same question must produce the same approved shape of answer every time. For internal campaign brainstorming, she opens it right up, because surprise is the product.
Same model, opposite settings, both correct. The dial isn't good or bad; it's task-fit. And run-to-run variation is never evidence that the machine "changed its mind": nothing in there has a mind to change.
Where the picture breaks
A raffle is pure chance. Here, the ticket allocation encodes everything the model learned, so the favourite is usually excellent, something no fair raffle could ever claim. The randomness is only in the draw, not in the weighting.
Also: "drawing" is one decoding method among several. Real systems select or sample from the distribution depending on their decoding settings (greedy top-pick, temperature sampling, top-p and more). Many tools don't expose any dial, and even "maximum consistency" settings aren't always perfectly identical run to run in deployed products. The concept is durable; the knobs vary by product.
Now make the call
In testing, Maya's client-facing assistant answered the same fee question three times with three differently-worded replies. This is a financial advice firm, and compliance wants approved, consistent wording every time. What does she do?
Try "ask the assistant why". It's the third appearance of the introspection trap, and the last time it should ever catch you.
Act 1 challenge: Friday, 4 pm
Board papers are due at 5. Your assistant's draft summary includes: “The Harwood Trust grant (applications close 30 September) totals $48,500.” Everything you've learned in Act 1, one decision at a time.
Apply it to your work
Sort this week's AI tasks into two piles: consistency tasks (board papers, compliance wording; turn variability down and supply an approved template) and ideation tasks (campaign angles, naming; open it up). Record both piles in your capstone checklist starter.
Your habit from this lesson
Match variability to the task: consistency tasks cold, ideation hot, and check whether your AI tool exposes the setting. Above all, stop reading run-to-run variation as lying, learning or mood. It's a raffle you can tune.
Go deeper (optional): the maths under the dial
The model's raw scores are turned into probabilities by a function called softmax. Temperature divides the scores before that conversion, which is why low values exaggerate gaps between candidates and high values flatten them. Products often combine temperature with top-p sampling: ignore the long tail of terrible candidates, and draw only from the smallest set covering, say, 90% of the probability.
Why isn't "temperature 0" always perfectly identical in real products? Deployed systems batch work across many users on parallel hardware, and tiny arithmetic-ordering differences can occasionally flip a near-tie. The concept, deterministic favourite-picking, holds; the engineering adds noise at the margins.
Deeper challenge: in the laboratory above, find the temperature at which the creative prompt's five-draw run first produces four different words. Then explain, in raffle terms, why the factual prompt won't do that at any temperature.
Back to your Lesson 1 note
In Lesson 1 you wrote your own explanation of what produces an AI answer. Now that you've seen the raffle: would you change anything? Most learners add something about the spread: answers aren't just "the most likely continuation" but a selection from a weighted field, tunable between consistent and varied.
Prove it to yourself
Two quick questions, untimed and retryable, and they count toward ★ Mastered (completed + decision + self-check all correct). This is what defeats the feeling of "that made sense" evaporating by next week.
Your field note
One or two sentences: why can the same question get different answers, and when would you want each behaviour?
✓ Lesson 4 stamped: Raffle Master, Act 1 complete
You tuned the dial, predicted the draws, and separated variability from unreliability. With that, you've finished Act 1: you now know what AI is actually doing. Act 2, how it learned to do it, starts with Lesson 5.
Evidence & review — how we know what this lesson claims
| Claim | Type | Basis | Review risk |
|---|---|---|---|
| Models produce a probability distribution over candidate tokens at each step | How it works | Primary research literature | Low |
| Decoding settings (temperature, top-p, greedy) reshape or replace sampling | How it works | Primary research and framework documentation | Low |
| Products layer sampling rules and often hide the dial | Product behaviour | Provider documentation | High — reviewed quarterly |
| "Temperature 0" is not always perfectly deterministic in deployed systems | How it works (empirical nuance) | Engineering analyses | Medium |
| The laboratory's distributions | Teaching device | Simulation, labelled; draws use real randomness deliberately | Medium |
| The raffle-drum analogy | Useful mental model | Limits stated in the lesson | Low |
Concept review due: January 2027. Product-behaviour claims: quarterly.
Where you are in the machine
You've reached the heart of the engine, and with your field note stamped, Act 1's layers light up. Select any layer for its role, lessons and your live progress.
The machine, bottom to top: training data (2) → token pieces (3) → prediction engine (1, this lesson, 5–9) → the desk (10–11) → senses (12) → live knowledge (13) → tools (14) → memory (15) → agent loops (16) → human judgement (17–21).
Next: how did it get this good?
You now know what the machine does: chop, weigh, raffle, repeat. Act 2 asks the deeper question: how did it learn to allocate those tickets so well? Nobody programmed the rules in. The story of how they got there is called training. Lesson 5: Ten Thousand Nudges.