Multimodality (teaching an LLM to see)

The How LLMs Work board Teaching It To See: a photograph cut into squares and slid into the middle of a sentence. This is how a machine that only reads got eyes, without anybody rebuilding it.

Before you start. From Tokens & Embeddings: a word never enters a language model as letters, it is swapped for a long list of numbers first, and every word’s list is the same length. That is all you need, because the model only ever works on those lists and cannot tell where one came from.