RAG from Zero · Episode 2
Embeddings: how AI knows “refund” = “paisa wapas”
Text becomes numbers that capture meaning. We build the idea from a 2-number toy map to 3,072 dimensions, then test it on DocBot's real data, where one model puts a Hinglish question in the wrong place.
The problem: zero letters in common
A customer asks DocBot: “paisa wapas kab milega?” The answer is in the handbook, but the handbook only says refund. The two phrases share zero letters.
In Episode 1 we tried plain Ctrl+F on the handbook:
| Search | Hits |
|---|---|
refund |
58 places in 7 docs |
paisa wapas kab milega? |
0 |
paisa / wapas / kab / milega |
0 / 0 / 0 / 0 |
Ctrl+F matches letters. Customers type meaning. Same intent, different words, different language, and keyword search sees nothing.
So we need a function: text in, numbers out, where the numbers tell us how close two texts are in meaning.
A hash that collides on purpose
Engineers already have a function that turns text into a number: a hash. But a hash does the opposite of what we want. refund and refunds differ by one letter, and a good hash throws them to completely different places. That is the whole point of a hash: tiny change in, big change out.
We want the reverse:
An embedding is a hash that collides on purpose. Same meaning → same place.
The 2-number meaning map
Start with the simplest version. Give every sentence just two numbers:
- How much is this sentence about money?
- How much is it about time?
| Sentence | Money | Time | Lands in |
|---|---|---|---|
| Refund kitne din mein aayega? | high | high | top-right |
| Paisa wapas kab milega? | high | high | top-right |
| Chhutti kitni milti hai? | low | high | a different corner |
The two refund questions land in the same corner even though they share no words. Now search is easy: turn the question into two numbers too, and pick the slips closest to it on the map.
Meaning has become a point. And between two points, you can measure a distance.
Why 1,536 dimensions and not 2
Two numbers are not enough. Two sentences can both be about money and time and still mean completely different things (a refund and a salary, for example). On a 2-number map they collide.
So real models give many numbers. Every extra number is one more direction in which meanings can separate. In 2 directions, “refund” and “salary” crash into each other; in 1,500 directions there is room for them to sit apart.
- Each number is called a dimension.
- The list of numbers for one text is its embedding (also called a vector).
- 1,536 is not magic. OpenAI’s
text-embedding-3-smallchose 1,536 dimensions; Google’sgemini-embedding-001gives 3,072.
Myth: each number means something
People imagine “number 7 = money”. Mostly, no. A single number on its own tells you nothing. The meaning lives only in which points are near which.
The map you see is a shadow
A screen can only show 2 directions. Every 2D “meaning map” (including ours below) is a projection, like the shadow of a 3D object on a wall. We used UMAP for ours. Things that are close usually stay close in the shadow, but some detail is lost. Never measure distances on the 2D picture; measure them on the full vectors.
Who sets the numbers? Training on pairs
Nobody sat down and assigned numbers to “refund”. The model learns them from data:
- Show the model huge numbers of pairs that mean the same thing: a question and its answer, a title and its page.
- Pull each same-meaning pair closer, like a spring.
- Push random pairs apart.
- Repeat millions of times.
The map builds itself. All the refund sentences end up in one neighbourhood. Nobody told the model what refund means; the pairs pulled it there. (This is called contrastive training.)
A consequence worth remembering: a model only knows the neighbourhoods its training pairs taught it. If it saw few Hinglish ↔ English pairs, it will not put them together. You will see exactly this below.
Cosine similarity: direction is meaning
How do we measure “close” on the map? Not with a ruler.
Picture every embedding as an arrow from the origin. Whether the arrow is long or short doesn’t matter. What matters is the direction. Two arrows pointing the same way = same meaning.
That is cosine similarity: the cosine of the angle between two arrows.
cos(a, b) = (a · b) / (|a| × |b|)
- 1 → pointing exactly the same way (same meaning)
- 0 → at right angles (unrelated)
Cosine vs dot product vs Euclidean
The first question every engineer asks: why cosine, and not dot product or Euclidean distance? We tried all three on DocBot’s real data.
- Dot product: multiply the two vectors number by number and add everything up. It counts both direction and length. Cosine is just the dot product with length removed.
- Euclidean distance: the straight-line distance between the two arrow tips (the ruler we said no to). Smaller distance = closer meaning.
The ranking code from docbot/src/e2c.py:
def rank_by(qv, pool_m, metric):
"""Indices into pool_m, best first, plus the score used."""
if metric == "cosine":
s = pool_m @ qv / (np.linalg.norm(pool_m, axis=1) * np.linalg.norm(qv))
return np.argsort(-s, kind="stable"), s
if metric == "dot":
s = pool_m @ qv
return np.argsort(-s, kind="stable"), s
s = np.linalg.norm(pool_m - qv, axis=1) # euclidean: smaller = closer
return np.argsort(s, kind="stable"), s
Result: all three gave the same full ranking of all 103 candidates, for both models and both queries. Top 5 for “paisa wapas kab milega?” on gemini-embedding-001:
| # | Cosine | Dot | Euclidean |
|---|---|---|---|
| 1 | paise wapas kaise milenge? 0.884 | same, 0.884 | same, 0.482 |
| 2 | refund kitne din mein aata hai? 0.821 | same | same, 0.598 |
| 3 | mujhe refund chahiye 0.763 | same | same, 0.688 |
| 4 | How long does a refund take? 0.716 | same | same, 0.753 |
| 5 | When will I get my refund? 0.711 | same | same, 0.760 |
Why identical? These models return unit vectors (every arrow has length 1). For unit vectors:
a · b = cos θ, so dot product is cosine.|a − b|² = 2 − 2·cos θ, so Euclidean distance goes down exactly as cosine goes up. Same order.
Our check of that identity on the saved vectors: max error 2.2 × 10⁻¹⁵, i.e. exact.
Practical rule: check the norm of one vector. If it’s 1, use dot product (it’s the cheapest) and stop worrying. The metric only matters for models that return raw, un-normalised vectors.
Matryoshka: cut 3,072 numbers down to 512
Bigger models give 3,072 numbers per slip: double 1,536, chosen by the model maker. More numbers, more directions, slightly better search. But storage grows too:
| Dimensions | Bytes per vector (float32) | 1 million vectors |
|---|---|---|
| 512 | 2 KB | 2 GB |
| 1,536 | 6 KB | 6 GB |
| 3,072 | 12 KB | 12 GB |
The trick is Matryoshka embeddings (named after Russian nesting dolls). Models trained this way pack the most important information into the first numbers. Cut off the tail, and it still works.
The cut is one line: keep the first d numbers, then re-normalise to length 1.
def unitise(m):
return m / np.linalg.norm(m, axis=1, keepdims=True)
SIZES = [64, 128, 256, 512, 768, 1024, 1536, 3072]
for d in [s for s in SIZES if s <= m.shape[1]]:
u = unitise(m[:, :d]) # keep first d numbers, back to length 1
e = e2b_measures(u)
Results on gemini-embedding-001, Hinglish query:
| Dims | Storage / 1M | Rank of “How long does a refund take?” | Refund items in top 10 | INV-2041 rank (of 10) |
|---|---|---|---|---|
| 64 | 256 MB | 7 | 10 | 4 |
| 128 | 512 MB | 7 | 10 | 2 |
| 256 | 1 GB | 5 | 10 | 1 |
| 512 | 2 GB | 4 | 10 | 1 |
| 1,536 | 6 GB | 4 | 10 | 1 |
| 3,072 | 12 GB | 4 | 10 | 1 |
At 512 dimensions (1/6 of the storage) we got the same answer as the full 3,072. What breaks first when you cut too far is exact-ID matching (INV-2041), not topic matching.
Notes:
- Truncation only works for models trained for it. OpenAI’s
text-embedding-3models take adimensionsparameter; Gemini takesoutput_dimensionality(supported 128 to 3,072; Google recommends 768, 1,536 or 3,072). Our 64-dim row is below Google’s minimum and is a stress test only. - If you truncate yourself, re-normalise. Both providers’ docs say so.
- Cutting dims can’t add knowledge.
text-embedding-3-small’s Hinglish miss (below) is there at every size.
Read the model card (query vs passage)
Some models want to be told which side is which. The e5 family expects a query: prefix on the question and passage: on every document. Its model card says that leaving them out degrades performance.
We tested it on intfloat/multilingual-e5-small. On our small pool the effect was small and mixed: 2 more first-place hits out of 33 English questions, slightly worse on Hinglish. The rule is real (it’s how the model was trained) and you should follow it, but it is not a magic fix.
Rule: before using any embedding model, read its model card once: dimensions, max input length, prefixes or task types, and whether vectors are normalised.
A free model on your laptop
multilingual-e5-small (384 dims, ~470 MB) ran locally on an M4 MacBook: about 8 seconds per 100 slips on the Apple GPU (MPS), $0, and the data never leaves the machine. English questions worked well (refund chunk at rank 2). But our Hinglish question landed in the wrong place again, next to pricing, leave and allowance questions. “Multilingual” on the label did not buy romanised Hindi.
DocBot’s real map
Theory done. We ran all of DocBot’s data through a real model, openai/text-embedding-3-small:
- 67 slips (chunks from 24 handbook docs) + 37 short sentences (refund questions in English and Hinglish, leave, invoices, pricing, HR, a few unrelated ones) = 104 points.
- Each one is 1,536 numbers, projected to 2D with UMAP.
The groups formed on their own: invoices on one side, HR policies on another, refunds on another. Nobody labelled them. That’s the springs at work.
Then we dropped in the question “paisa wapas kab milega?”. The code (from common.py and e2.py):
def cosine_top(qvec, mat, k):
q = np.asarray(qvec, dtype=float)
m = np.asarray(mat, dtype=float)
sims = m @ q / (np.linalg.norm(m, axis=1) * np.linalg.norm(q))
order = np.argsort(-sims)[:k]
return [(int(i), float(sims[i])) for i in order]
top = cosine_top(mat[qi], mat[pool], 10)
Output:
paisa wapas kab milega?:
1 0.7940 sentence paise wapas kaise milenge?
2 0.5678 sentence galti se do baar paisa kat gaya
3 0.4654 sentence refund kitne din mein aata hai?
4 0.4523 sentence mujhe refund chahiye
5 0.4351 sentence internet allowance kitna hai?
6 0.4015 sentence chhutti kitni milti hai?
7 0.3626 sentence maternity leave kitne hafte ki hai?
8 0.3580 sentence Starter plan kitne ka hai?
9 0.3306 sentence invoice ka PDF kaise download karun?
10 0.3053 chunk refund_policy_2025.md#3
The top 9 are all Hinglish, including internet allowance, leave and pricing. The first English refund slip is at rank 10. And the direct comparison is brutal:
| Pair | Cosine |
|---|---|
| “paisa wapas kab milega?” vs “How long does a refund take?” | 0.190 |
| “paisa wapas kab milega?” vs “internet allowance kitna hai?” | 0.435 |
For this model, language weighs more than meaning.
Every model draws its own map
So do embeddings fail? No. This is this model’s map, and every model draws its own map.
Two consequences:
- You can’t mix vectors from two models. They live in different spaces.
- If you change models, you must re-embed every slip.
Same slips, same question, now on a multilingual model, google/gemini-embedding-001. In the code, only the model name changes:
MODELS = {
"openai": ("openai/text-embedding-3-small", "openai_text-embedding-3-small"),
"gemini": ("google/gemini-embedding-001", "google_gemini-embedding-001"),
}
model, slug = MODELS[sys.argv[1]]
vecs, cost = gateway.embed(texts, "e2m_gemini", model=model)
| Rank | text-embedding-3-small | cos | gemini-embedding-001 | cos |
|---|---|---|---|---|
| 1 | paise wapas kaise milenge? | 0.794 | paise wapas kaise milenge? | 0.884 |
| 2 | galti se do baar paisa kat gaya | 0.568 | refund kitne din mein aata hai? | 0.821 |
| 3 | refund kitne din mein aata hai? | 0.465 | mujhe refund chahiye | 0.763 |
| 4 | mujhe refund chahiye | 0.452 | How long does a refund take? | 0.716 |
| 5 | internet allowance kitna hai? | 0.435 | When will I get my refund? | 0.711 |
| 6 | chhutti kitni milti hai? | 0.402 | I want my money back | 0.707 |
| 7 | maternity leave kitne hafte ki hai? | 0.363 | refund_policy_2026.md#3 (chunk) | 0.697 |
| 8 | Starter plan kitne ka hai? | 0.358 | money back | 0.696 |
| 9 | invoice ka PDF kaise download karun? | 0.331 | refund_policy_2025.md#3 (chunk) | 0.695 |
| 10 | refund_policy_2025.md#3 (chunk) | 0.305 | refund | 0.671 |
Every one of gemini’s top 10 is about refunds, Hinglish and English together. The first English refund sentence moved from rank 13 to rank 4. On gemini’s map, the question sits in one cluster with all 7 English refund sentences, all 4 other Hinglish refund lines and the refund chunks.
Embeddings didn’t fail. The map was from the wrong model.
Don’t compare scores across models
Look at the unrelated sentence “best biryani in Hyderabad”:
| Model | cos(query, biryani) |
|---|---|
| text-embedding-3-small | 0.160 |
| gemini-embedding-001 | 0.483 |
| multilingual-e5-small | 0.804 |
Each model has its own score range. 0.716 on gemini is not “3.8× more similar” than 0.190 on OpenAI. Compare ranks, and gaps above that model’s own floor, never raw scores across models. This also means a fixed threshold like “keep anything above 0.5” must be re-tuned for every model.
Twist: INV-2041 vs INV-2047
A customer asks: “What’s the status of INV-2041?”
To an embedding model, INV-2041 and INV-2047 are almost the same text:
| Model | cos(“INV-2041”, “INV-2047”) |
|---|---|
| text-embedding-3-small | 0.980 |
| gemini-embedding-001 | 0.903 |
Ranking DocBot’s 10 invoice chunks against “INV-2041” on text-embedding-3-small:
INV-2041 vs invoice chunks: [('invoice_INV-2048.md#0', 0.4014), ('invoice_INV-2040.md#0', 0.4001),
... ('invoice_INV-2041.md#0', 0.3555)]
All 10 scores sit between 0.36 and 0.40, and the right invoice came 10th of 10, dead last. Gemini ranked it first, but by only 0.011 over INV-2042. A narrow win, not a fix.
IDs, SKUs, error codes and version numbers have no “meaning”, so meaning search can’t tell them apart. The fix is keyword search (BM25) alongside embeddings, which comes later in this course.
Which embedding model should you use?
Not “whatever is top of the leaderboard”. The leaderboard is not your data.
One example from the BRIGHT paper (Su et al., ICLR 2025): SFR-Embedding-Mistral, then top of MTEB, scored 59.0 on the MTEB retrieval subset but only 18.3 on BRIGHT, a reasoning-heavy retrieval benchmark.
How to choose:
- Write ~50 real questions your users ask, in the languages they actually use (Hinglish counts as its own language here).
- For each, mark the slip that should come back.
- Embed your corpus with 2–3 candidate models and measure how often the right slip is in the top k.
- Weigh in price, dimensions/storage, and whether it can run locally.
We’ll build exactly this 50-question evaluation for DocBot later in the course.
Cheat sheet
- Embedding = a list of numbers that places text by meaning. A hash that collides on purpose.
- Dimension = one number in that list. 1,536 and 3,072 are model choices, not magic.
- Individual numbers mean nothing; only neighbours mean something.
- Models learn the map from same-meaning pairs (contrastive training). They only know what their pairs taught them.
- Cosine similarity measures direction. On unit vectors, cosine = dot product, and Euclidean gives the same ranking.
- Matryoshka models can be cut (e.g. 3,072 → 512) with almost no loss. Re-normalise after cutting.
- Every model draws its own map: never mix vectors, re-embed on model change, never compare raw scores across models.
- Test on your languages. Our Hinglish question failed on two models and worked on one.
- Embeddings are bad at exact IDs. Pair them with keyword search.
- Read the model card: prefixes, max length, normalisation.
Next episode: vector databases. How do you find the nearest slip among millions in milliseconds?
Sources
- Kusupati et al., Matryoshka Representation Learning, NeurIPS 2022. arXiv:2205.13147
- Su et al., BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval, ICLR 2025. arXiv:2407.12883
- OpenAI embeddings guide: normalised vectors, the
dimensionsparameter. - Gemini API embeddings docs:
output_dimensionality, normalisation below 3,072. - intfloat/multilingual-e5-base model card:
query:/passage:prefixes. - All DocBot numbers were measured by us on 2026-11-06 and 2026-11-15. Total paid API spend for this episode’s tests: under one US cent.