What are embeddings?
A scroll-driven lesson with a live word map &semantic search that matches without sharing a word.
A token id is just a number.
Last lesson turned your text into token ids.
But the id itself has no meaning.
1642 is simply a label. The number does not tell the model what the token means.
So the model needs another way to represent meaning.
It does this by giving each token a long list of numbers.
Give every word a position.
Imagine every word as a point on a large map.
Similar words appear close together.
Cats may sit near dogs. Paris may sit near Rome.
The model learns these positions from how words are used in text.
Humans do not manually place the words. The positions are learned during training.
Words that are closer are usually more similar.
Once words have positions, we can compare them.
Words used in similar ways tend to have similar positions.
The model can measure this similarity using numbers.
Try a few examples below.
Real models often compare the angle between two positions. The basic idea is still the same.
These positions can even capture relationships.
The space can learn patterns between words.
A famous example uses king, man, woman, and queen.
Move from king away from man. Then move toward woman.
The result often lands close to queen.
Similar patterns can appear for countries and capitals, or singular and plural words.
Search no longer needs the exact same words.
Normal keyword search looks for matching words.
That can fail when two phrases mean the same thing but use different words.
For example, “time off” and “annual leave” are related.
Embeddings can place them close together even when the words are different.
This idea is used in semantic search and many chat with documents systems.
Embeddings can also learn human biases.
The model learns patterns from human written text.
That means it can also learn unfair or harmful associations.
The model is not told to create these patterns.
They can appear because they were present in the training data.
This matters when embeddings are used in hiring, search, recommendations, or other important systems.
The real space has hundreds of dimensions.
The examples above use a simple map so we can see what is happening.
Real embeddings use hundreds or even thousands of numbers for each token.
We cannot draw that many dimensions on a screen.
But the same ideas still work. Position, distance, and direction all matter.
These embeddings are the input for the next major step: attention.
Search that understands
Match by meaning instead of spelling. Synonyms, paraphrases and other languages all just work.
Recommendations
Things near each other are things the same person tends to want. No rules required.
Clustering
Group ten thousand support tickets by what they are about, without reading any of them.
Cheap classification
Embed once, train a tiny model on top. Often beats fine-tuning for a fraction of the cost.
The input to everything
Attention compares positions. Without embeddings a transformer has nothing to compare.
Inherited bias
Learned from our text, including the parts we would not defend. It arrives with the usefulness.
How embeddings are actually trained — the objective that pulls similar words together, which deserves its own lesson. Also contextual embeddings, the significant upgrade where a word’s position depends on the sentence around it rather than being fixed in a table; that is what modern models really use, and it is what lets “bank” mean two things. And the practical craft: which embedding model to choose, how to chunk documents before embedding them, and why similarity scores from different models cannot be compared. None of that changes the core picture — meaning as position, similarity as distance, relationships as directions.