Chopping Language into Lego
See your own sentence the way the machine sees it — as snap-together pieces — and discover why AI miscounts letters.
By the end you'll be able to
- Explain that models process text as tokens — frequent chunks of characters — not letters or whole words.
- Predict why letter-level tasks (counting, reversing, rhyming) can fail.
- Recognise that everything AI reads and writes is measured — and priced — in tokens.
- Predict
- Explore
- Decide
- Field note
⚡ 30-second warm-up — From Lesson 2: the assistant answered a niche funder's rules as fluently as general grant practice. What was the danger?
Thin coverage in the diet, delivered in the same fluent voice — thin topics get plausible boilerplate, so they're verify-first zones.
Start with a famous failure
Ask an AI assistant how many letter r's are in "strawberry", and it may get it wrong. This is a machine that can draft a shareholder letter. Why can't it count to three?
Commit to a prediction
Why would a fluent writing machine fail at counting letters?
This lesson changes one thing: you'll stop thinking "it reads my words" and start thinking "it reads numbered pieces — spelling is inference, and everything I send and receive is metered in tokens."
The one idea in this lesson
Computers work with numbers, so text must become numbers. One number per letter would be wasteful; one per word would break on every new name. So builders create a catalogue of a few tens of thousands of frequent character chunks. "Understanding" might be two chunks; a rare surname might be five. Each chunk gets a number.
These chunks — tokens — are what the model actually reads and predicts. It usually receives token representations rather than a convenient letter-by-letter grid — numbered pieces, not spelled-out words.
This explains real behaviour: "berry" is predicted as one piece, so counting the r's inside it is like counting the studs inside a sealed Lego brick. The model can often infer spelling from patterns it has seen — but letter-exact tasks remain brittle, which is precisely the signature of a representation problem rather than a skill problem. It also explains why AI limits and prices are quoted in tokens: tokens are the machine's native unit — and your budget's.
frontier models extend the same trick beyond text: images arrive as grids of patch-pieces and audio as slices, all joining text tokens in one stream (Lesson 12, on the map below). Hidden "reasoning" steps are tokens too — metered and billed like everything else. A few research systems skip tokens for raw characters, but today's mainstream — and today's pricing — is token-shaped.A picture to hold on to
Lego bricks, moulded from the most common chunks of language. "Straw" and "berry" might each be one brick. You can build anything from the bricks — but you can never see inside one.
- The bricks = tokens.
- The brick catalogue = the vocabulary, fixed before training.
- Building = generating, one brick at a time.
- Sealed bricks = no native access to the letters inside.
See your own words the machine's way
Type anything — your name, workplace jargon, another language — and watch it get chopped. This page uses a simplified illustrative tokeniser; real models' catalogues differ, but the behaviour you'll see is the real phenomenon. Nothing you type leaves this page.
Where this bites at work
Kerry, an allied health practitioner, pastes a long clinical guideline into an assistant and hits a limit — measured not in pages but in tokens. Her first instinct is that the tool is broken. It isn't: she's met the machine's native unit.
Tokens are how AI tools meter everything: input limits, output length, and price. Learning to think in tokens is learning to read the meter — Lesson 10 turns that into a management skill, and it's an optimisation skill, not a "keep everything short" rule.
Where the picture breaks
Real Lego bricks are identical and interchangeable. Tokens are not: each carries learned relationships to other tokens — "berry" arrives soaked in associations with fruit, jam and summer. (How meaning attaches to pieces is Lesson 7's beautiful story.)
Also: unlike Lego, there's no standard catalogue. Different models chop the same text differently — token counts, and therefore costs, vary between providers. And some newer systems work at the character level; tokens describe today's mainstream, not a law of nature.
Now make the call
Kerry pastes an 80-page clinical guideline into her assistant and asks for a summary of what changed in the screening recommendations. The reply arrives in seconds and reads beautifully. Before she updates her clinic's protocols — what does she do?
Try the "did you read it all?" option — it's the same trap you met in Lesson 1, and it will keep reappearing in the wild.
Apply it to your work
Find the document type your team pastes into AI tools most often — grant guidelines, policy briefs, meeting packs. Check your tool's input limit, and set a team rule: split long documents into sections rather than trusting a silent truncation.
Your habit from this lesson
When AI fails a spelling, counting or precise-format task, recognise it as a tokenisation limit and restructure rather than retry: ask it to separate the letters first, or hand exact character work to a tool that's built for it. Diagnosing the failure type — instead of concluding "AI is dumb" or "AI is magic" — is the professional move.
Go deeper (optional): how the catalogue gets built, and who pays more
The catalogue comes from byte-pair encoding (BPE), which you can think of in three steps: start with single characters; repeatedly merge the most frequent adjacent pair into a new catalogue entry; stop when the catalogue reaches its size budget. Frequency decides everything — which is why common English words become single tokens while rare words shatter.
A consequence with an equity edge: languages under-represented in the training diet get chopped into more, smaller pieces. The same sentence can cost several times more tokens in some languages than in English — same meaning, higher price and tighter limits.
Deeper challenge: take a phrase from your own workplace jargon and predict its chop before running it through the tokeniser above. Repeat until you've correctly predicted one rare-word shatter.
Prove it to yourself
Two quick questions — untimed, retryable, and they count toward ★ Mastered (completed + decision + self-check all correct). This is what defeats the feeling of "that made sense" evaporating by next week.
Your field note
One or two sentences: what does the machine actually see when it reads your writing, and what does that explain?
✓ Lesson 3 stamped: Brick Reader
You diagnosed a famous AI failure from its mechanism instead of its vibes. That's the competence this stamp records.
Evidence & review — how we know what this lesson claims
| Claim | Type | Basis | Review risk |
|---|---|---|---|
| Mainstream models process text as subword tokens built by frequency-based algorithms (e.g. BPE) | How it works | Primary research literature | Low |
| Letter-exact tasks remain brittle; spelling is often inferable | How it works (empirical) | Behavioural evaluations | Medium — improving with reasoning models |
| Under-represented languages cost more tokens for the same meaning | How it works (empirical) | Tokeniser analyses | Medium — verify per model |
| AI products meter limits and pricing in tokens | Product behaviour | Provider pricing documentation | High — reviewed quarterly |
| This page's tokeniser | Teaching device | Simplified illustration, labelled; real catalogues differ | Medium |
| The sealed-Lego-brick analogy | Useful mental model | Limits stated in the lesson | Low |
Concept review due: January 2027. Product-behaviour claims: quarterly.
Where you are in the machine
Tokens are the machine's reading layer. Select any layer for its role, lessons and your live progress.
The machine, bottom to top: training data (2) → token pieces (this lesson) → prediction engine (1, 4–9) → the desk (10–11) → senses (12) → live knowledge (13) → tools (14) → memory (15) → agent loops (16) → human judgement (17–21).
Next: the moment of choice
You now know the pieces. But at each step the machine must pick which piece comes next — and that choice is a weighted raffle you can actually adjust. Lesson 4: The Dice Inside the Machine.