Singapore-based practical guides, tutorials and experiments in AI, computing, modelling, simulation, optimisation and quantum computing, with research notes and hands-on workflows.
Word Embeddings Explained: Math Behind LLMs & Vector AI
Word embeddings map language to vectors. But raw distance skews semantics; cosine similarity is essential. Explore geometric proofs and cosine formulas →
A guide to word embeddings — where meaning becomes mathematics, and vectors do the talking.
Quick Answer & Key Insight
How Do Word Embeddings Turn Words into Math?
Yes, word embeddings map words to numeric vectors. While raw Euclidean distance measures geometric separation, cosine similarity isolates semantic angle regardless of magnitude, enabling vector arithmetic like vec("King") − vec("Man") + vec("Woman") ≈ vec("Queen"). See the complete mathematical formulas and comparison tables below →
When a search engine retrieves a document about automobiles in response to a query about cars, it is not matching text character by character. Somewhere beneath the interface, the system understands that these two words are semantically related. The mechanism behind that understanding is the word embedding — and once you see the geometry, you cannot unsee it.
This article walks through the key mathematical operations that make embeddings work: distance, similarity, arithmetic, scaling, and the dot product. Each concept is illustrated with concrete numerical vectors so the math is visible, not just described. Real embeddings typically use hundreds of dimensions; the 3- and 4-dimensional examples here preserve all the structure while staying readable on a page.
1 · What is a Word Embedding?
A word embedding is a representation of a word as a vector — an ordered list of numbers — in a high-dimensional space. A typical embedding model might use 300 dimensions, so the word cat becomes a point with 300 coordinates. That sounds abstract, but the key insight is this: the position of that point encodes meaning.
This is what researchers call a semantic space. Words with related meanings end up positioned close to each other. King and Queen live near each other. Paris and London live near each other. Bicycle and democracy live far apart. The model learns these positions not from human-curated rules, but from the statistical patterns of how words appear together in enormous text corpora.
EXAMPLE: 4-DIMENSIONAL VECTORS (simplified from real 300-dim embeddings)
The first three dimensions encode royalty, authority, and age. The fourth dimension encodes gender: positive = masculine, negative = feminine.
Think of it as a map where the geography is meaning. Every word is a pin, and the distances between pins reflect semantic relationships rather than physical ones.
2 · The Geometry of Meaning: Distance and Similarity
Once words are points in space, we need a way to measure how close they are. Two approaches dominate: Euclidean distance and cosine similarity. For the examples below, we use a 3-dimensional temperature embedding:
In practice, NLP systems almost universally prefer cosine similarity over Euclidean distance. It ignores the length of vectors entirely and focuses only on the angle between them — two vectors pointing the same direction score 1.0 regardless of their magnitude.
COSINE SIMILARITY
Comparison Expression
Vector Magnitude & cos(θ)
Dot Product Result (v · v)
A Little Loud
Magnitude: 0.539 · cos(θ) = 1.000
0.29
Very Loud
Magnitude: 1.616 · cos(θ) = 1.000
2.61
Table 2: Dot product and magnitude comparison for loudness vectors.
Both words are perfectly collinear — cosine similarity is 1.0 in both cases. But the dot products are 0.29 vs 2.61, a 9× difference. This is why recommendation systems and attention mechanisms in transformer models often prefer raw dot products: when you want to know not just whether a document is relevant but also how prominently it discusses a topic, the dot product gives you both dimensions at once.
6 · Practical Applications
Search engines convert your query into a vector and retrieve documents whose vectors are nearest to it in the semantic space — using cosine similarity to rank by relevance regardless of exact word match. When you search for car insurance and the engine returns results about vehicle coverage, it is doing nearest-neighbour lookup in embedding space, exactly as the Hot/Warm/Cold example in Section 2 demonstrates.
Recommendation systems represent your interests as a vector computed from your history, then find products whose vectors are closest to yours. The dot product is particularly useful here: a highly-relevant item with a large magnitude — analogous to Very Loud — will score higher than a mildly-relevant item even if they point in the same direction.
Large language models use the scaled dot product directly inside the attention mechanism. For every token, a query vector and a set of key vectors are compared via dot product to determine which parts of the context deserve attention — a direct descendant of the arithmetic explored in Section 5.
Quick Reference: Embedding Operations
Operation
Formula
Section 2-5 Result
Euclidean Distance
√( Σ (ai − bi)2 )
d(Hot,Warm) = 0.346 d(Hot,Cold) = 2.163
Cosine Similarity
(a·b) / (‖a‖×‖b‖)
cos(Hot,Warm) = +0.998 cos(Hot,Cold) = -0.499
Vector Arithmetic
a ± b
King-Man+Woman → nearest Queen (d = 0.400)
Scalar Multiplication
λ · a
Large × 2 → near Gigantic Loud ÷ 2 → near Soft
Dot Product
a·b = Σ aibi
cos = 1.00 for both; dot 0.29 (soft) vs 2.61 (loud)
Table 3: Summary of vector mathematics and geometry operations in embedding spaces.
📚 Foundational Math • Recommended Reading:
Once embeddings convert words and queries into continuous geometry, language models use softmax normalization and multi-class cross-entropy to evaluate vocabulary distributions and token probabilities. For a complete mathematical derivation of how probabilistic models are optimized, why lower loss is strictly better, and why confident wrong predictions trigger explosive penalties, read our companion guide: Multi-Class Log Loss Explained: Math, MLE & Python Code.
Leave a Reply