How Vector Databases Work: Embeddings, ANN, and HNSW

A vector database is a database whose primary query is “find me the things most similar to this one,” where similarity is measured in high-dimensional vector space rather than by exact field matches. It is the storage layer that makes semantic search, retrieval-augmented generation, recommendation, and a lot of modern AI features possible. If you have used chat-with-your-docs, a shopping search that understands “cozy fall jacket,” or a coding assistant that pulls up the relevant part of your codebase, a vector database was almost certainly doing the finding underneath.

Understanding how vector databases work comes down to three questions: how do you turn real things into vectors, how do you search millions of them for the nearest ones fast, and how do you keep that search useful when you also need to filter, update, and scale. The first is embeddings. The second is approximate nearest neighbor search, and it is where all the interesting algorithms live. The third is systems engineering.

What a vector database is, and is not

A traditional database answers exact queries: give me the rows where status equals paid. A vector database answers similarity queries: give me the rows whose meaning is closest to this query. The rows are still there, with all their normal fields, but each row also carries a vector (a list of numbers) and the database knows how to compare vectors by distance. The comparison is the whole point: two texts with similar meaning produce vectors that are close together, so finding relevant content becomes a geometry problem.

This makes a vector database complementary to, not a replacement for, a relational database. Most real systems are hybrid: a vector column for semantic similarity and ordinary columns for exact filters and joins. This is why pgvector, a Postgres extension, is so popular: you get both in one engine.

Embeddings: turning data into vectors

Before anything can be searched, it has to be embedded. An embedding model takes a piece of content (text, an image, audio) and outputs a fixed-length vector, often somewhere between 384 and 4096 dimensions. The model is trained so that items a human would call similar land near each other in this space. The result is that semantic relationships become spatial ones: related concepts cluster, analogies show up as parallel directions, and “nearest” starts meaning “most relevant.”

The embedding step happens outside the database. You run your documents through an embedding model, store the resulting vectors in the database alongside the source text and metadata, and at query time you embed the query with the same model and search for nearby vectors. The embedding model defines the coordinate system; the database just does geometry inside it. This is why you can never mix embedding models: a vector from one model is meaningless in another’s space.

The search problem: nearest neighbors in high dimensions

The core query is: given a query vector, return the k stored vectors closest to it by some distance measure, usually cosine similarity or Euclidean distance. Conceptually this is trivial: compute the distance to every vector, sort, take the top k. That is exact brute-force search, and for a few thousand items it is fine.

It does not scale. Comparing a query against a million 768-dimensional vectors is hundreds of millions of floating-point operations per query, and it has to happen in tens of milliseconds. Exact search is O(n) in the index size, which is acceptable up to maybe a hundred thousand items and painful beyond it. This is the central problem vector databases exist to solve: how to find nearest neighbors approximately, but fast, with a tunable trade-off between speed and accuracy.

Approximate nearest neighbor search

The solution is approximate nearest neighbor, or ANN, search. ANN algorithms trade a small, controllable amount of accuracy for large speedups by building an index structure that prunes the search space, so you never compare the query against every vector. Instead of scanning everything, you navigate a structure that gets you close quickly and then refine. The catch is that you might occasionally miss a true neighbor (the recall is not 100 percent) but in practice a well-tuned ANN index reaches 95 to 99 percent recall at a fraction of the brute-force cost.

Every modern vector database is built on one or more ANN index types. Three families dominate, and understanding them at a high level is most of understanding vector databases.

Graph-based indexes: HNSW

Hierarchical Navigable Small World graphs build a multi-layer graph where each vector is a node connected to its nearby neighbors. Search starts at a coarse top layer and greedily hops toward the query, dropping to finer layers to refine. HNSW is the default in most vector databases because it is fast, accurate, and handles incremental inserts well. Its cost is memory: the graph structure is large, and the whole index usually lives in RAM.

Cluster-based indexes: IVF

Inverted File indexes partition vectors into clusters using a method like k-means, then store which cluster each vector belongs to. At query time you find the nearest cluster centroids and search only within those clusters, ignoring the rest. IVF is simple and memory-efficient, and its accuracy is tunable by how many clusters you probe. It pairs well with quantization.

Compression: product quantization

Product quantization compresses vectors into compact codes so the index fits in less memory and distance computation is cheaper, at the cost of precision. It is rarely used alone (you usually see IVF with PQ, or HNSW with PQ) to combine fast navigation with small footprint. Quantization is the reason you can search a hundred million vectors on one machine instead of a cluster.

These compose. A typical production index is something like IVF-PQ-HNSW: cluster to prune, quantize to compress, graph-navigate within clusters for speed. The database exposes the knobs (how many clusters to probe, how aggressively to quantize) and you tune them to hit the recall and latency you need.

Metadata filtering and hybrid search

Nearest-neighbor alone is rarely enough. Real queries also say things like “only documents from 2025” or “only products in electronics.” Vector databases support metadata filtering so you can combine a similarity search with exact constraints on other fields. The subtle part is when to filter: pre-filtering shrinks the search space before ANN, which can hurt recall if it removes too much; post-filtering runs ANN first and then drops non-matching results, which can return too few. Engines differ in how they handle this, and it is a common performance and correctness trap.

Hybrid search goes further by blending vector similarity with keyword search like BM25. Vector search captures meaning; keyword search captures exact terms. The two fail in different ways (a vector search misses exact product codes, a keyword search misses paraphrases) so fusing their rankings with a method like reciprocal rank fusion gives you the best of both. Most serious retrieval systems are hybrid.

Updates, CRUD, and scale

A database is not just a search engine; it has to support inserts, updates, and deletes over time. This is harder than it looks for ANN indexes. HNSW handles inserts gracefully but deletes leave tombstones that eventually need compaction. IVF is cheaper to build but re-clustering on large updates is expensive. Real systems separate a fast mutable layer for recent writes from an immutable optimized index that gets rebuilt periodically, merging the two at query time. This is the same write-amp/read-amp trade-off that every database navigates, just applied to vector indexes.

Scaling past one machine adds sharding, where vectors are partitioned across nodes and a query fans out to the relevant shards. Some engines shard by metadata so filtered queries only hit one shard; others shard randomly and query all shards. Replication, consistency, and cost all flow from these choices, and they are the main differentiators between managed vector databases.

How RAG and agents use vector databases

The dominant use case today is retrieval-augmented generation. Documents are chunked, embedded, and stored; at query time the question is embedded and the nearest chunks are retrieved as context for a language model. The vector database is the retrieval stage of RAG, and its recall and latency are a large part of RAG quality.

Agents use them for memory. An AI agent that wants to recall past interactions or stored facts embeds each one and queries the vector store for the most relevant memories before its next step. Long-running agents would be lost without this: the context window is too small to hold everything, so semantic recall over a vector store is how they remember.

Choosing a vector database

The field has consolidated around a few solid options, and the right choice depends mostly on what you already run. pgvector adds vector search to PostgreSQL and is the right default if you already use Postgres, up to tens of millions of vectors. Pinecone and Weaviate are fully managed and remove operations at a price premium. Milvus and Qdrant are open-source engines built for scale and self-hosting. OpenSearch and Elasticsearch add vector search to mature search platforms with strong hybrid and filtering support. The ANN algorithms underneath are largely the same: the decision is about managed versus self-hosted, scale, and how well it integrates with your existing data.

Pro Tips
Tune recall, do not assume it. Every ANN index has a speed-accuracy knob. Measure recall against brute-force search on a sample of queries and tune until you hit 95 percent or higher before you ship. Untuned indexes silently return worse results and you will blame the model.
Measure latency at the percentile, not the average. Vector search has a long tail. Track p95 and p99 latency, not just the mean, and budget for the tail or users will see occasional slow queries that the average hides.
Re-embed when you switch models, version your embeddings. Changing the embedding model changes the coordinate system. Store the model name and version with every vector, and re-embed the whole index when you upgrade, or similarity scores become garbage.