Vector Databases

A data engineer's guide to vector databases: when they beat traditional search, how to evaluate options, and operational considerations.

A vector database is an index over embedding space. If you understand how a B-tree index works in Postgres, you understand why a vector index (HNSW, IVFFlat) exists and what its trade-offs are. The concepts translate directly: write amplification, index build time, precision vs. recall trade-offs.

The most important architectural decision is whether you need a dedicated vector database or whether your existing data platform can handle it. Postgres with pgvector handles millions of vectors comfortably. Snowflake and BigQuery have native vector search. Start with what you have before adding operational complexity.

When you do need a dedicated store, evaluate on: latency at your target percentile (p99, not average), index build time (matters for freshness), filtering support (metadata pre-filters before vector similarity), and your team's ability to operate it. Pinecone is fully managed; Qdrant, Weaviate, Milvus are self-hosted options.

Operational hygiene matters more than people expect. Vectors go stale when the underlying embedding model changes -- a model upgrade requires a full re-index. Plan for this from day one. Version your vectors alongside your embedding model version.

Namespace your collections by data domain and update cadence, not by team. A collection that mixes documents updated hourly with documents updated monthly will cause freshness and relevance problems. Partition by lifecycle.