New Senses: Multimodal AI
By the end you'll be able to
- Explain how images, audio and documents get converted into token-like pieces that join text in the same meaning-space.
- Predict characteristic multimodal failures: fine text in images, counting objects, spatial precision.
- Choose the best input for a task: a photo, a transcript, or plain text.
- Predict
- Test senses
- Decide
- Field note
⚡ 30-second warm-up: from Lesson 7, what does "distance" mean on the machine's meaning-map?
How alike two things are used, not how alike they look or sound. Today, photos and audio join words on that very same map.
Show an AI a photo of your fridge and it suggests dinner. Cameras don't speak, so what has to happen first?
Start with an odd fact
The prediction machine only ever deals in tokens, pieces of text. Yet it can describe a photograph, transcribe a voice memo, and read a scrawled note. Nothing about the core engine changed. Something happens before the photo ever reaches it.
Commit to a prediction
Before the prediction machine can "see" your fridge photo, the photo must be turned into…
This lesson changes one thing: you'll stop picturing an AI "looking" at a photo the way you do. Every medium, whether image, audio, or document, gets repacked into the same coordinate system words already live on. Then the same familiar loop runs.
Discover the mechanism
The prediction machine only eats tokens. So multimodal systems add converters at the door. An image gets cut into a grid of patches, and each patch is encoded into coordinates on the same meaning-atlas where words live (Lesson 7's embedding space). Audio gets sliced and treated the same way. From there, the familiar loop (Lesson 9) runs. Attention lets a question like "what's expiring in this fridge?" spotlight the relevant patches, and the answer emerges token by token.
The conversion is lossy and pattern-based, and that predicts the classic failures: tiny text in photos misread, objects miscounted (patches, not an inventory), left/right confusions, and confident description of things that aren't quite there. New senses, same machine, same core caution. A plausible description isn't a verified observation.
image generation is a related but separate mechanism (diffusion: sculpting noise toward a description). It's deliberately kept out of this lesson's core; the Go deeper section draws the line clearly.Every kind of cargo, such as crates, barrels, envelopes, gets repacked into standard shipping containers before entering the country. Inside the border, the road network handles only containers and never knows or cares what port they came from.
- Cargo types = image, audio, text. Repacking = encoders converting each medium into token-like pieces.
- Standard containers = embeddings in the shared meaning-space (Lesson 7). The road network = the same transformer pipeline (Lessons 8–9).
- Where it breaks: containers preserve contents exactly; this repacking is lossy and interpretive, a summary of visual features, not the pixels themselves.
Test its senses yourself
This uses six worked examples with a stated ground truth, not a live model. For each scene, predict whether it will nail the task, then reveal what it actually said.
Your prediction accuracy:
Where this bites at work
Kerry photographs referral letters for intake summaries. It works well on typed letterheads, but it's unreliable on GPs' handwritten margins. So she keeps human checking for handwriting, and reserves photographing for typed pages only. Task-fit judgement in action.
Her rule: match the medium to the task. Photograph for gist and description. Transcribe or type anything precision-critical. Always verify extracted numbers.
"The AI sees the image like I do" is the misconception to retire. It encodes a lossy statistical summary. There's no inner picture being looked at, and no lived visual experience behind the description, however confident that description sounds.
Now make the call
Kerry has a stack of referral letters to summarise before this afternoon's clinic. Some are typed, some have handwriting in the margins. What does she do?
Try asking it to double-check its own reading too. It's worth seeing that a confident "yes, I'm sure" is just another generated line.
Apply it to your work
Look at how your team feeds images or scans to AI tools. For each recurring case, decide: photograph for gist, or type/transcribe for anything precision-critical (numbers, names, handwriting, fine print). Write the rule down before the next mistake, not after.
Prove it to yourself
Two quick questions, untimed and retryable. They count toward ★ Mastered (completed + decision + self-check all correct). This is what stops "that made sense" evaporating by next week.
Explain it in your own words
Write two or three sentences for a colleague. Then compare your note against the rubric below. It checks ideas, not wording, and only you see what you write.
✓ Lesson 12 stamped: Sense Tester
You predicted successes and failures across six real cases before seeing the answers. Lesson 13 asks: its knowledge still stops at training day, so what if it could look things up?
Evidence & review — how we know what this lesson claims
| Claim | Type | Basis | Review risk |
|---|---|---|---|
| Images, audio and documents are converted into token-like pieces sharing the model's meaning-space | How it works | Architecture documentation | Low |
| Multimodal systems characteristically struggle with fine text, exact counts and spatial precision | Product behaviour | Documented evaluation results | Medium — reviewed quarterly |
| Image generation (diffusion) is mechanistically distinct from image understanding | How it works | Primary research literature | Low |
| Specific model capability levels for image/audio tasks | Product behaviour | Provider evaluation reports | High — reviewed quarterly |
| The six worked sense-tester examples | Teaching device | Illustrative, ground truth stated, labelled non-live | Medium |
| The customs-checkpoint analogy | Useful mental model | Limits stated in the lesson | Low |
Concept review due: January 2027. Product-behaviour claims: quarterly.
Where you are in the machine
The machine has gained senses on top of the desk and the briefing. Select any layer to see its role, its lessons, and your live progress.
The machine, bottom to top: training data (2) → token pieces (3) → prediction engine (1, 4–9) → the desk (10–11) → senses (12) → live knowledge (13) → tools (14) → memory (15) → agent loops (16) → human judgement (17–21).
Next: giving it a library card
Senses widen what the machine can take in, but its knowledge still stops at training day. Next: giving it a library card.