{"slug":"vectors-embeddings-and-search","title":"Vectors, embeddings, and search: an intuition-first guide","date":"2026-02-19","content":"\n## Vectors, embeddings, and search\n\nWhen we talk about AI, search, or LLMs, it often sounds like magic. *You type words. Something understands them. It finds meaning, not just keywords.* But under the hood, there’s no magic. There are vectors. Lots of them. \n\nNot symbols, not rules, not hidden dictionaries. Just numbers arranged in space. \n\nWhen I built my previous blog, I used vectors to power *article recommendations* (surfacing related content based on what you'd just read). I never really went deep on how any of it worked. This article is me fixing that.\n\nNo heavy math. Just mental models you can actually feel.\n\n## Text isn’t what models read\n\nIt’s tempting to imagine models reading raw text, but they don’t. Language is broken down into smaller pieces called *tokens*.\n\nYou might wonder why this step even matters. Bear with me, it'll make sense in a minute.\n\n\n\nAt this stage, nothing understands anything. Just segmentation. Slicing and indexing. It's the bridge between language and math.\n\n## From ids to vectors\n\nAn integer id isn’t useful on its own. It’s just an index. What matters is what it points to. Each token retrieves a vector from a large learned table, *the embedding matrix*.\n\n\n\nEvery row in this matrix contains hundreds or thousands of values. Those values weren't typed in by a human. They were learned during training, adjusted over billions of examples. \n\nAnd here's the important part: this isn't a neural computation happening in real time. It's a lookup. When a token appears, we fetch its vector. Token appears, vector comes out. That's it.\n\n## Everything starts as coordinates\n\nImagine a blank 2D space. An infinite canvas. Every point can be described with two numbers:\n\n$$\n(x, y)\n$$\n\nThat’s a vector. Nothing fancy. Just coordinates.\n\n\n\nIf you squint a bit, a vector is just a container for numbers. On its own, it means nothing. Meaning only appears when we decide what those numbers represent.\n\nIn graphics, they might be positions. In physics, forces. In machine learning, concepts. And here’s the part that matters.\n\nThe $(x, y)$ version is just the toy model, real embeddings don’t live in two dimensions. If you're thinking *“okay but that's too simple to be useful“*, you're right. Here's what real embeddings actually look like:\n\n$$\n(x_1, x_2, x_3, \\dots, x_{1536})\n$$\n\nThousands of coordinates. No human can visualize that space. But the math still works exactly the same: *add, compare, measure distance, measure angle*.\n\nEverything that follows in this article is just different ways of using those numbers.\n\n## Turning meaning into position\n\nImagine we define two axes. *Cold and hot* on the x-axis. *Sad and happy* on the y-axis. Words like *ice*, *fire*, *party*, and *funeral* suddenly find a natural position in space. Not because we wrote definitions for them but because their coordinates reflect relationships. And suddenly, position means something.\n\n\n\nNow, an important clarification: real embedding models don't have axes named *hot* or *happy* or whatever. Their dimensions are abstract, unlabeled. But the idea is the same, just not in 2D.\n\n## Comparing meaning geometrically\n\nOnce everything is represented as vectors, comparison becomes math. One of the most common tools is cosine similarity. It measures the angle between two vectors:\n\n$$\n\\begin{gathered}\n\\text{similarity}(A, B) = \\cos(\\theta) \\\\\n\\text{where } \\theta \\text{ is the angle} \\\\\n\\text{between the two vectors}\n\\end{gathered}\n$$\n\nWhich in practice becomes:\n\n$$\n\\begin{gathered}\n\\frac{A \\cdot B}{\\|A\\| \\|B\\|} \\\\\n\\text{where } A \\cdot B = \\sum_i A_i B_i\n\\end{gathered}\n$$\n\nIf vectors are normalized, cosine similarity is just a dot product.\n\n$$\nA \\cdot B\n$$\n\nYeah, that looks like a lot. Don't stress it. What matters is the outcome. If vectors point in similar directions, the score is high. If they diverge, the score drops. If they're perpendicular, the similarity is close to zero.\n\n\n\nThis single operation is behind semantic search, recommendations, RAG. One operation. That's all it takes. Similarity isn't stored, it's computed.\n\n## From intuition to real embeddings\n\nAs we said, real embeddings aren’t two numbers. They’re more like:\n\n$$\n[0.12, -0.93, 0.44, \\ldots, 0.003]\n$$\n\n*Hundreds. Sometimes thousands*. You can’t point at one number and say, “that’s meaning”. No single dimension tells the story. It's the combination that matters. A bit like how no single pixel makes an image, but zoom out and suddenly there's a face.\n\nWe don’t label these axes. We don’t interpret them directly. We just use them.\n\nHere’s the important shift. We don’t compare text anymore. We compare distance. Two vectors that are close together represent similar meaning. That's what semantic search runs on.\n\n\n\nLet's make it concrete. Imagine you have a database full of documents: docs, blog posts, code snippets. Each one has already been converted into a vector and stored. \n\nA user searches: *“How to fix slow react app?“*. That query gets embedded into a vector too. Now we just compute the distance between that vector and every stored vector.\n\nClosest matches win. No keyword matching. No handcrafted rules. Just geometry. That's the whole thing, really.\n\n## Where transformers enter\n\nUp to this point, we’ve been a bit hand-wavy. We’ve talked about vectors, distances, similarity, as if those vectors simply existed. They don't. They are produced. You might be asking: *“okay, but who produces them?“* That's exactly the right question. And that's where *transformers* enter the picture.\n\nThis section was the hardest to write. Transformers involve a lot of moving parts, and most explanations either go too deep or hand-wave everything. I tried to find the middle ground.\n\n### From static vectors to contextual ones\n\nWhen a sentence enters a model, each token starts with a vector from the [embedding matrix](#from-ids-to-vectors). At that moment, the vector is static. The word *“react”* always maps to the same initial coordinates. No context. No nuance. That's a problem.\n\nBecause *react* in *“react app“* doesn’t mean the same thing as *react* in *“How humans react to stress“*. Static embeddings can’t capture that difference. Transformers can.\n\n### Attention changes that\n\nTransformers introduce *attention*. Instead of treating tokens independently, each token looks at the others and asks: *Which parts of this sentence matter to me right now?*\n\nIn practice, attention can be fully bidirectional like in encoder models, or strictly causal like in GPT-style decoders. But the intuition is the same either way.\n\nThe answer isn’t binary. It’s weighted. Some tokens pull strongly. Others barely matter. You can think of attention as contextual gravity. Words pulling on each other to reshape their positions in space.\n\n\n\nEach layer slightly reshapes the representation. Not by adding new information, but by passing context around. After one layer, a token knows a bit about its neighbors.\n\nAfter several layers, it knows: what role it plays, what it refers to, how it relates to the rest of the sentence.\n\nBy the final layer, it's not *“react“* anymore. It's *“react“* in *“How to fix slow react app?“*\n\n## Bringing it all together\n\nWe start with text. We end with spatial relationships.\n\n```mermaid\nflowchart TB\n subgraph Input[\"Input\"]\n direction LR\n A[\"Text\"]\n B[\"Tokenization\"]\n C[\"Token IDs\"]\n end\n subgraph Static_Embedding[\"Static Embedding\"]\n direction LR\n D[\"Embedding Matrix\"]\n E[\"Static Token Vectors\"]\n end\n subgraph Transformer_Core[\"Transformer Core\"]\n direction LR\n F[\"Transformer Layers\"]\n G[\"Contextual Vectors\"]\n end\n subgraph Embedding_Model[\"Embedding Model\"]\n direction TB\n H[\"Pooling / Projection\"]\n I[\"Final Embedding Vector\"]\n end\n subgraph LLM_Model[\"LLM Model\"]\n direction TB\n J[\"Linear Projection to Vocabulary\"]\n K[\"Next Token Prediction\"]\n end\n A --> B\n B --> C\n D L_D_E_0@--> E\n F --> G\n H L_H_I_0@--> I\n J L_J_K_0@--> K\n Input L_Input_Static_Embedding_0@--> Static_Embedding\n Static_Embedding L_Static_Embedding_Transformer_Core_0@--> Transformer_Core\n Transformer_Core L_Transformer_Core_Embedding_Model_0@--> Embedding_Model & LLM_Model\n\n\n L_D_E_0@{ curve: natural } \n L_H_I_0@{ curve: natural } \n L_J_K_0@{ curve: natural } \n L_Input_Static_Embedding_0@{ curve: natural } \n L_Static_Embedding_Transformer_Core_0@{ curve: natural } \n L_Transformer_Core_Embedding_Model_0@{ curve: natural } \n L_Transformer_Core_LLM_Model_0@{ curve: natural }\n```\n\nSearch, retrieval, generation, it all builds on this. Simple math, quietly powering most of what modern AI does.\n\n## Closing thought\n\nVectors aren't just a technical detail buried somewhere in the stack. They're what everything runs on. Once you start seeing language as geometry, a lot of things start making sense. RAG, semantic search, agents, recommendations, it's all the same idea, just applied differently.\n\nIf you've used any of these systems without thinking too hard about what's underneath, hopefully this gave you a clearer picture of what's actually going on.\n\n*Nicolas*\n\n\n\t\n\t\tThe original Transformer stack is from Vaswani et al., [*Attention Is All\n\t\tYou Need*](https://arxiv.org/abs/1706.03762) (NeurIPS 2017). This post stays\n\t\tat intuition level; that paper is the standard reference for scaled\n\t\tdot-product attention and the original Transformer stack.\n\t\n\n","description":"A hands-on dive into vectors, embeddings, and how modern semantic search actually works."}