Tokens and embeddings
A model cannot read letters. How many pieces does your sentence turn into - and why is that number different in another language?
Your sentence arrives at the model in pieces, and each chip on the page is one piece with the real id it would be handed. The slider sets how many pieces the tokenizer knows. Watch the same sentence fall from one chip per character to one chip per word.
Board: Tokens and Embeddings →A text box, a slider and four panels. On a narrow screen the panels stack. Nothing plays automatically.
try this Drag How many pieces it knows left to right, then open the tab Another script, and an emoji, and watch How many tokens.
Included with a plan
This lab is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
see also Why AI Needs a Sense of Smell, Large AI Models Are Cultural and Social Technologies, Attention Is All You Need and Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks in Papers.