Retrieval-augmented generation demos beautifully and fails quietly. The gap between the two is almost never the model - it's the retrieval pipeline around it.
Chunking is a product decision
Naive fixed-size chunks split tables mid-row and separate answers from their context. Chunk along the document's own structure - headings, sections, list boundaries - and store the heading path with each chunk. A chunk that knows it lives under "Refunds > Enterprise plans" retrieves far better than an orphaned paragraph.
Hybrid retrieval beats pure vectors
Embeddings miss exact identifiers - SKUs, error codes, names. Combine vector similarity with classic keyword search (BM25) and merge results. Add a lightweight re-ranker on the top 20 candidates and answer quality jumps visibly.
- Retrieve wide (20-50 candidates), re-rank narrow (top 5 into the prompt)
- Store metadata filters - date, product, audience - and apply them before similarity
- Show sources in the answer; it builds trust and exposes retrieval bugs
Evaluate retrieval separately from generation
Build a golden set of 50-100 real questions with known source passages. Measure whether retrieval finds the right passage at all - if it doesn't, no prompt engineering can save the answer. Track this number in CI like a test suite.
Freshness is a pipeline, not a feature
Stale indexes are the most common silent failure. Re-embed on document change events, not on a monthly cron, and version your index so you can roll back a bad ingestion the way you'd roll back a deploy.