Images, video and sound are natural inputs. Text is a compression format: a lossy, symbolic encoding of perceived reality that the reader must decompress back into pictures, sounds and feelings.
Large language models are trained almost entirely on that compressed layer.
1. They learn the map, not the territory
An LLM knows "red" only through sentences people wrote about red. It has never seen the colour, felt heat, or heard a violin. Its knowledge is second-hand: a reconstruction of experience from descriptions of experience.
2. Text throws away most of the signal
A photo or a voice clip carries tone, timing, layout, texture and emotion all at once. A sentence flattens all of it. Things that are obvious to humans, like how a heavy box slides or how close a cup is to the table's edge, are rarely written down, because everyone already knows them.
So LLMs are strong at articulated knowledge and weaker at tacit, embodied knowledge. This is Moravec's paradox again: what is easy for humans is hard for machines, and the reverse.
3. Even the text interface is artificial
Models don't read letters or words. They read subword tokens, chunks chosen by a compression algorithm. That is why "how many r's are in strawberry?" can trip them up. The input format is an engineering artifact, not a natural unit of perception.
4. Multimodal is the foundation, not an add-on
Video and audio are far denser sources of grounding. A few seconds of footage contains physics, causality, intent and social dynamics, all of which text has to spell out in words. This is why vision, audio, robotics and world-model research matter: they let a system learn from the raw stream, with text as one interface on top.
5. And yet, text works astonishingly well
Text may be unnatural for perception, but it is the best tool humans have for abstract thought: logic, mathematics, code, planning, and passing knowledge across generations. Language is already a high-level abstraction of human cognition, so training on it gave models a shortcut to reasoning without first solving perception.
The takeaway
LLMs are trained on humanity's compressed output, and they are remarkably good at working inside that compression. The next step toward general intelligence is probably to recover what was lost: learning from raw sensory streams the way a child does, rather than only from the written description of the world.
Text will not disappear. Humans give machines instructions in text and similar formats, so text stays the interface on top. But it should sit on top of perception, not stand in for it. Multimodal learning is the foundation: it supplies the grounding that gives words their meaning, and text is the abstraction and interface layer built over it. A model that only has the text layer can talk fluently about the world. A multimodal model can actually connect those words to what they refer to.