RAG Pipelines in Production: What the Tutorials Don't Tell You

Retrieval Quality Sets the Ceiling

A generation model cannot say what it was not given. Every quality problem we have chased in a deployed RAG system traced back to the retrieval stage, and upgrading the model was the last lever we pulled, not the first.

The Tuning That Pays Off

In order of impact per engineering hour, based on what we have shipped and measured.

• Chunk on document structure rather than a fixed token window
• Keep headings, tables, and section paths in the indexed text
• Add a reranker before touching the prompt or the model
• Blend lexical and vector search for exact terms, codes, and names
• Store the source version so answers can be reproduced later

What Breaks After Launch

The corpus changes constantly, and the pipeline is usually designed as if it does not.

• Deleted documents still surfacing from a stale index
• Silent embedding-model upgrades that invalidate every stored vector
• One document type dominating results after a bulk import
• No signal for "the answer is not in the corpus" questions
• Evaluation suites frozen on the day the project shipped