AI
positional encoding
added so order is known
Source board
This definition comes from the board below. Open it to see the term in context.
Tokens and EmbeddingsAIAlso defined on Attention and the Transformer.
More terms from this board
- analogy arithmeticking−man+woman≈queen
- BPEbyte-pair encoding (merge frequent pairs → vocab)
- cosine similaritycloseness by angle between vectors
- dense vectorall-numbers representation (vs one-hot)
- dimension dsize of the vector (768…12288)
- embeddinga token's learned vector (d dims)
- embedding matrixthe vocab×d lookup table (learned)
- semantic searchnearest-vector retrieval (RAG, l.08)
- static vs contextual embeddingfixed per word vs context-shifted
- tokenthe atomic unit a model reads (a subword)
- token IDan integer index
- tokenizertext ↔ token IDs (a fixed pre-step)
- vector databasestores/searches embeddings
- vector spacewhere meaning lives geometrically
- vocabularythe ~50-100k known tokens