AI
tokenizer
text ↔ token IDs (a fixed pre-step)
Source board
This definition comes from the board below. Open it to see the term in context.
Tokens and EmbeddingsAIMore terms from this board
- analogy arithmeticking−man+woman≈queen
- BPEbyte-pair encoding (merge frequent pairs → vocab)
- cosine similaritycloseness by angle between vectors
- dense vectorall-numbers representation (vs one-hot)
- dimension dsize of the vector (768…12288)
- embeddinga token's learned vector (d dims)
- embedding matrixthe vocab×d lookup table (learned)
- positional encodingadded so order is known
- semantic searchnearest-vector retrieval (RAG, l.08)
- static vs contextual embeddingfixed per word vs context-shifted
- tokenthe atomic unit a model reads (a subword)
- token IDan integer index
- vector databasestores/searches embeddings
- vector spacewhere meaning lives geometrically
- vocabularythe ~50-100k known tokens