30 sec3 min coredeep dive optional
A vector database stores embeddings and answers one query at scale: find the items most similar to this one, using approximate nearest neighbor indexes such as HNSW.
- Embeddings turn text, images, and audio into vectors where distance means semantic similarity.
- ANN indexes trade a small, tunable amount of recall for orders of magnitude faster search.
- RAG and agents rely on this retrieval layer to ground answers in your content.
A vector database is a database whose primary query is “find me the things most similar to this one,” where similarity is measured in high-dimensional vector space rather than by exact field matches. It is the storage layer that makes semantic search, retrieval-augmented generation, recommendation, and a lot of modern AI features possible. If you have used chat-with-your-docs, a shopping search that understands “cozy fall jacket,” or a coding assistant that pulls up the relevant part of your codebase, a vector database was almost certainly doing the finding underneath.
Understanding how vector databases work comes down to three questions: how do you turn real things into vectors, how do you search millions of them for the nearest ones fast, and how do you keep that search useful when you also need to filter, update, and scale. The first is embeddings. The second is approximate nearest neighbor search, and it is where all the interesting algorithms live. The third is systems engineering.
What a vector database is, and is not
A traditional database answers exact queries: give me the rows where status equals paid. A vector database answers similarity queries: give me the rows whose meaning is closest to this query. The rows are still there, with all their normal fields, but each row also carries a vector (a list of numbers) and the database knows how to compare vectors by distance. The comparison is the whole point: two texts with similar meaning produce vectors that are close together, so finding relevant content becomes a geometry problem.
This makes a vector database complementary to, not a replacement for, a relational database. Most real systems are hybrid: a vector column for semantic similarity and ordinary columns for exact filters and joins. This is why pgvector, a Postgres extension, is so popular: you get both in one engine.
Embeddings: turning data into vectors
Before anything can be searched, it has to be embedded. An embedding model takes a piece of content (text, an image, audio) and outputs a fixed-length vector, often somewhere between 384 and 4096 dimensions. The model is trained so that items a human would call similar land near each other in this space. The result is that semantic relationships become spatial ones: related concepts cluster, analogies show up as parallel directions, and “nearest” starts meaning “most relevant.”
The embedding step happens outside the database. You run your documents through an embedding model, store the resulting vectors in the database alongside the source text and metadata, and at query time you embed the query with the same model and search for nearby vectors. The embedding model defines the coordinate system; the database just does geometry inside it. This is why you can never mix embedding models: a vector from one model is meaningless in another’s space.
The search problem: nearest neighbors in high dimensions
The core query is: given a query vector, return the k stored vectors closest to it by some distance measure, usually cosine similarity or Euclidean distance. Conceptually this is trivial: compute the distance to every vector, sort, take the top k. That is exact brute-force search, and for a few thousand items it is fine.
It does not scale. Comparing a query against a million 768-dimensional vectors is hundreds of millions of floating-point operations per query, and it has to happen in tens of milliseconds. Exact search is O(n) in the index size, which is acceptable up to maybe a hundred thousand items and painful beyond it. This is the central problem vector databases exist to solve: how to find nearest neighbors approximately, but fast, with a tunable trade-off between speed and accuracy.
Approximate nearest neighbor search
The solution is approximate nearest neighbor, or ANN, search. ANN algorithms trade a small, controllable amount of accuracy for large speedups by building an index structure that prunes the search space, so you never compare the query against every vector. Instead of scanning everything, you navigate a structure that gets you close quickly and then refine. The catch is that you might occasionally miss a true neighbor (the recall is not 100 percent) but in practice a well-tuned ANN index reaches 95 to 99 percent recall at a fraction of the brute-force cost.
Every modern vector database is built on one or more ANN index types. Three families dominate, and understanding them at a high level is most of understanding vector databases.
Graph-based indexes: HNSW
Hierarchical Navigable Small World graphs build a multi-layer graph where each vector is a node connected to its nearby neighbors. Search starts at a coarse top layer and greedily hops toward the query, dropping to finer layers to refine. HNSW is the default in most vector databases because it is fast, accurate, and handles incremental inserts well. Its cost is memory: the graph structure is large, and the whole index usually lives in RAM.
Cluster-based indexes: IVF
Inverted File indexes partition vectors into clusters using a method like k-means, then store which cluster each vector belongs to. At query time you find the nearest cluster centroids and search only within those clusters, ignoring the rest. IVF is simple and memory-efficient, and its accuracy is tunable by how many clusters you probe. It pairs well with quantization.
Compression: product quantization
Product quantization compresses vectors into compact codes so the index fits in less memory and distance computation is cheaper, at the cost of precision. It is rarely used alone (you usually see IVF with PQ, or HNSW with PQ) to combine fast navigation with small footprint. Quantization is the reason you can search a hundred million vectors on one machine instead of a cluster.
These compose. A typical production index is something like IVF-PQ-HNSW: cluster to prune, quantize to compress, graph-navigate within clusters for speed. The database exposes the knobs (how many clusters to probe, how aggressively to quantize) and you tune them to hit the recall and latency you need.
Why does approximate search sometimes miss the true nearest neighbor, and when does that matter?
Every ANN index makes the same bargain: instead of comparing the query against every vector, it navigates a structure that visits only the neighborhoods likely to contain close matches. HNSW walks a layered graph of links between vectors; IVF first narrows the space to a few clusters and scans inside them. In both cases the true nearest neighbor can sit outside the regions the search explored, so the index returns an excellent answer rather than a guaranteed best one. The recall metric measures exactly this gap: what fraction of the true top results the index actually finds.
Whether the gap matters depends on what the search feeds. For recommendations and discovery, a 95 percent recall setting is invisible to users, because the difference between the best match and the fourth best match is rarely meaningful. For retrieval augmented generation the stakes are higher but still forgiving in practice: the model needs relevant context, and any of the top handful of passages usually provides it. The cases that genuinely demand exact search are small and structurally different, such as deduplication where near duplicates must be found deterministically, or compliance lookups over a bounded corpus where brute force is cheap enough.
The practical discipline is to measure rather than assume. Run a sample of real queries against both brute force and the tuned index, compute recall, and pick the operating point that keeps p95 latency inside budget. Revisit after large corpus growth or embedding model changes, because both shift where the speed recall frontier sits. Write the measurement down and keep it next to the deployment configuration, because a recall number without the query sample and index parameters that produced it cannot be reproduced, and an unreproducible recall target is a hope rather than a budget.
Metadata filtering and hybrid search
Nearest-neighbor alone is rarely enough. Real queries also say things like “only documents from 2025” or “only products in electronics.” Vector databases support metadata filtering so you can combine a similarity search with exact constraints on other fields. The subtle part is when to filter: pre-filtering shrinks the search space before ANN, which can hurt recall if it removes too much; post-filtering runs ANN first and then drops non-matching results, which can return too few. Engines differ in how they handle this, and it is a common performance and correctness trap.
Hybrid search goes further by blending vector similarity with keyword search like BM25. Vector search captures meaning; keyword search captures exact terms. The two fail in different ways (a vector search misses exact product codes, a keyword search misses paraphrases) so fusing their rankings with a method like reciprocal rank fusion gives you the best of both. Most serious retrieval systems are hybrid.
Updates, CRUD, and scale
A database is not just a search engine; it has to support inserts, updates, and deletes over time. This is harder than it looks for ANN indexes. HNSW handles inserts gracefully but deletes leave tombstones that eventually need compaction. IVF is cheaper to build but re-clustering on large updates is expensive. Real systems separate a fast mutable layer for recent writes from an immutable optimized index that gets rebuilt periodically, merging the two at query time. This is the same write-amp/read-amp trade-off that every database navigates, just applied to vector indexes.
Scaling past one machine adds sharding, where vectors are partitioned across nodes and a query fans out to the relevant shards. Some engines shard by metadata so filtered queries only hit one shard; others shard randomly and query all shards. Replication, consistency, and cost all flow from these choices, and they are the main differentiators between managed vector databases.
How RAG and agents use vector databases
The dominant use case today is retrieval-augmented generation. Documents are chunked, embedded, and stored; at query time the question is embedded and the nearest chunks are retrieved as context for a language model. The vector database is the retrieval stage of RAG, and its recall and latency are a large part of RAG quality.
Agents use them for memory. An AI agent that wants to recall past interactions or stored facts embeds each one and queries the vector store for the most relevant memories before its next step. Long-running agents would be lost without this: the context window is too small to hold everything, so semantic recall over a vector store is how they remember.
Choosing a vector database
The field has consolidated around a few solid options, and the right choice depends mostly on what you already run. pgvector adds vector search to PostgreSQL and is the right default if you already use Postgres, up to tens of millions of vectors. Pinecone and Weaviate are fully managed and remove operations at a price premium. Milvus and Qdrant are open-source engines built for scale and self-hosting. OpenSearch and Elasticsearch add vector search to mature search platforms with strong hybrid and filtering support. The ANN algorithms underneath are largely the same: the decision is about managed versus self-hosted, scale, and how well it integrates with your existing data.
How do managed vector services compare with embedded engines in practice?
The managed option moves operational burden out of your codebase: backups, version upgrades, index rebuilding across replicas, and horizontal scaling become the vendor’s problem, and the API usually adds filtering, hybrid search, and monitoring on top. The embedded option keeps data next to the application, which removes a network hop from every query and makes the deployment story trivial for small systems. Libraries built on HNSW or similar indexes run comfortably inside the same process as the rest of a prototype, which is exactly why they dominate the prototype stage.
The decision variables are scale, freshness, and team shape. Below a few million vectors with modest update rates, an embedded engine is often the honest answer, and graduating later is a migration rather than a rewrite. Once updates arrive continuously from many writers, or the corpus grows past what one node holds comfortably, replication and index maintenance start dominating the embedded setup, and a managed service earns its cost. Teams with strict data residency requirements sometimes split the difference: embeddings live in the existing primary database using its native vector type, and the application accepts a lower ceiling in exchange for keeping one system to govern.
Whatever the choice, keep the embedding model name and version beside every vector, and keep the index rebuild scripted. The migration cost that actually bites teams is not the move between engines; it is the re-embed of a large corpus after an unversioned model change.
How do embedding model choices shape the database design around them?
The embedding model determines vector dimension, distance metric behavior, and storage cost, so it belongs in the design conversation from day one. Higher dimensional models carry more nuance per item but inflate index memory and slow each graph hop; quantization and dimensionality reduction claw some of that back at a small recall cost. Pick the metric the model was trained for, because cosine rankings from a model tuned for dot product are subtly wrong in ways that surface as vague misses rather than errors.
The deeper commitment is versioning. A corpus embedded with one model lives in that coordinate system, so switching models means re-embedding everything and rebuilding every index, not translating vectors. Store the model name and version beside each vector, keep re-embed scripts rehearsed, and treat model upgrades as planned migrations with measured recall before and after.
Pro TipsTune recall, do not assume it. Every ANN index has a speed-accuracy knob. Measure recall against brute-force search on a sample of queries and tune until you hit 95 percent or higher before you ship. Untuned indexes silently return worse results and you will blame the model.
Measure latency at the percentile, not the average. Vector search has a long tail. Track p95 and p99 latency, not just the mean, and budget for the tail or users will see occasional slow queries that the average hides.
Re-embed when you switch models, version your embeddings. Changing the embedding model changes the coordinate system. Store the model name and version with every vector, and re-embed the whole index when you upgrade, or similarity scores become garbage.
Further reading
Tune recall, do not assume it. Every ANN index has a speed-accuracy knob. Measure recall against brute-force search on a sample of queries and tune until you hit 95 percent or higher before you ship. Untuned indexes silently return worse results and you will blame the model.
Measure latency at the percentile, not the average. Vector search has a long tail. Track p95 and p99 latency, not just the mean, and budget for the tail or users will see occasional slow queries that the average hides.
Re-embed when you switch models, version your embeddings. Changing the embedding model changes the coordinate system. Store the model name and version with every vector, and re-embed the whole index when you upgrade, or similarity scores become garbage.
Vector databases are the storage half of how RAG works and the memory half of how AI agents work. For the model that produces the embeddings, see our LLM foundations. The algorithms underneath (HNSW, IVF, product quantization) are decades of research compressed into a few well-engineered engines, and they are the reason semantic search For a vendor’s own tour of the category, Pinecone’s vector database explainer is a useful second opinion.