Embeddings

What data engineers need to know about embeddings: model selection, generation pipelines, storage, and drift detection.

An embedding is a lossy compression of meaning into a fixed-size vector. Every choice you make about how to generate embeddings -- which model, what input text, whether to include metadata, whether to fine-tune -- is a choice about what meaning to preserve and what to discard.

Model selection matters less than people think, until it matters a lot. text-embedding-3-small (OpenAI) or embed-english-v3.0 (Cohere) cover most use cases. The time to evaluate alternatives is after you have an eval harness showing your baseline retrieval quality -- not before.

Embedding generation is a batch pipeline. Treat it as one. Design for idempotency: if a document has not changed, do not re-embed it. Store embeddings alongside a hash of the source content. Run generation in parallel with rate limiting. Log failures with enough context to retry specific records.

Embedding drift is the silent killer of RAG systems. If the embedding model changes -- even a minor version bump -- your stored vectors and any new vectors are no longer in the same space. Search quality degrades silently. Pin your embedding model version, track it as a hard dependency, and build a re-embedding pipeline you can run when you upgrade.

Fine-tuning embeddings is almost never the right first move. The gain over a general-purpose model is real but modest for most domains. The maintenance burden (keeping the fine-tuned model operational, re-fine-tuning on new data) is significant. Spend that effort on better chunking and retrieval first.

Dimensionality reduction (PCA, UMAP) before storage can cut costs by 4-8x with minimal quality loss. Worth evaluating if you are storing tens of millions of vectors.