New Senses: Multimodal AI
By the end you'll be able to
- Explain that images, audio and documents are converted into token-like pieces that join text in the same meaning-space.
- Predict characteristic multimodal failures: fine text in images, counting objects, spatial precision.
- Choose when a photo, a transcript or plain text is the better input for a task.
- Predict
- Test senses
- Decide
- Field note
⚡ 30-second warm-up — From Lesson 7: what does "distance" mean on the machine's meaning-map?
How alike two things are used, not how alike they look or sound. Today, photos and audio join words on that very same map.
Show an AI a photo of your fridge and it suggests dinner. Cameras don't speak — so what has to happen first?
Start with an odd fact
The prediction machine only ever deals in tokens — pieces of text. Yet it can describe a photograph, transcribe a voice memo, and read a scrawled note. Nothing about the core engine changed. Something happens before the photo ever reaches it.
Commit to a prediction
Before the prediction machine can "see" your fridge photo, the photo must be turned into…
This lesson changes one thing: you'll stop picturing an AI "looking" at a photo the way you do, and start seeing every medium — image, audio, document — repacked into the same coordinate system words already live on, before the same familiar loop runs.
Discover the mechanism
The prediction machine only eats tokens. So multimodal systems add converters at the door: an image is cut into a grid of patches, each patch encoded into coordinates on the same meaning-atlas where words live (Lesson 7's embedding space); audio is sliced and treated similarly. From there, the familiar loop (Lesson 9) runs — attention lets a question like "what's expiring in this fridge?" spotlight the relevant patches, and the answer emerges token by token.
The conversion is lossy and pattern-based, which predicts the classic failures: tiny text in photos misread, objects miscounted (patches, not an inventory), left/right confusions, and confident description of things that aren't quite there. New senses, same machine, same core caution: plausible description is not verified observation.
image generation is a related but separate mechanism (diffusion — sculpting noise toward a description), deliberately kept out of this lesson's core; this lesson's Go deeper draws the line clearly.Every kind of cargo — crates, barrels, envelopes — gets repacked into standard shipping containers before entering the country. Inside the border, the road network handles only containers and never knows or cares what port they came from.
- Cargo types = image, audio, text. Repacking = encoders converting each medium into token-like pieces.
- Standard containers = embeddings in the shared meaning-space (Lesson 7). The road network = the same transformer pipeline (Lessons 8–9).
- Where it breaks: containers preserve contents exactly; this repacking is lossy and interpretive — a summary of visual features, not the pixels themselves.
Test its senses yourself
Conceptual illustration — six worked examples with a stated ground truth, not a live model. For each scene, predict whether it will nail the task, then reveal what it actually said.
Your prediction accuracy:
Where this bites at work
Kerry photographs referral letters for intake summaries: great on typed letterheads, unreliable on GPs' handwritten margins — so she keeps human checking for handwriting, and reserves photographing for typed pages only. Task-fit judgement in action.
The rule she uses: match the medium to the task — photograph for gist and description, transcribe or type for precision-critical detail, and always verify extracted numbers.
"The AI sees the image like I do" is the misconception to retire. It encodes a lossy statistical summary; there is no inner picture being looked at, and no lived visual experience behind the description — however confident that description sounds.
Now make the call
Kerry has a stack of referral letters to summarise before this afternoon's clinic — some typed, some handwritten in the margins. What does she do?
Try asking it to double-check its own reading too — worth seeing that a confident "yes, I'm sure" is just another generated line.
Apply it to your work
Look at how your team feeds images or scans to AI tools. For each recurring case, decide: photograph for gist, or type/transcribe for anything precision-critical (numbers, names, handwriting, fine print). Write the rule down before the next mistake, not after.
Prove it to yourself
Two quick questions — untimed, retryable, and they count toward ★ Mastered (completed + decision + self-check all correct). This is what defeats the feeling of "that made sense" evaporating by next week.
Explain it in your own words
Two or three sentences for a colleague. Then compare against the rubric — it checks ideas, not wording, and only you see your note.
✓ Lesson 12 stamped: Sense Tester
You predicted successes and failures across six real cases before seeing the answers. Lesson 13 asks: its knowledge still stops at training day — what if it could look things up?
Evidence & review — how we know what this lesson claims
| Claim | Type | Basis | Review risk |
|---|---|---|---|
| Images, audio and documents are converted into token-like pieces sharing the model's meaning-space | How it works | Architecture documentation | Low |
| Multimodal systems characteristically struggle with fine text, exact counts and spatial precision | Product behaviour | Documented evaluation results | Medium — reviewed quarterly |
| Image generation (diffusion) is mechanistically distinct from image understanding | How it works | Primary research literature | Low |
| Specific model capability levels for image/audio tasks | Product behaviour | Provider evaluation reports | High — reviewed quarterly |
| The six worked sense-tester examples | Teaching device | Illustrative, ground truth stated, labelled non-live | Medium |
| The customs-checkpoint analogy | Useful mental model | Limits stated in the lesson | Low |
Concept review due: January 2027. Product-behaviour claims: quarterly.
Where you are in the machine
The machine has gained senses on top of the desk and the briefing. Select any layer for its role, lessons and your live progress.
The machine, bottom to top: training data (2) → token pieces (3) → prediction engine (1, 4–9) → the desk (10–11) → senses (12) → live knowledge (13) → tools (14) → memory (15) → agent loops (16) → human judgement (17–21).
Next: giving it a library card
Senses widen what the machine can take in — but its knowledge still stops at training day. Next: giving it a library card.