Your RAG answer sounds plausible. Did retrieval find the right thing?

A wrong RAG answer can come from ingestion, query rewriting, retrieval, or unfaithful generation. Evaluation must separate those stages.

Build an evidence-backed question set

Sample real queries and label answerability, supporting passages, and acceptable answers. Include questions absent from the corpus to test abstention.

Score retrieval and generation separately

For retrieval measure evidence recall, noise, and rank. For generation measure faithfulness, completeness, and citation alignment. Ragas defines useful metrics, but automated scoring still needs human calibration.

Fix by failure category

Missing evidence points to ingestion, chunking, or search. Correct evidence with a wrong answer points to prompting, ordering, or model behavior. Stale sources require lifecycle fixes, not a larger model.

Pre-release checks

• The set includes unanswerable questions
• Citation correctness is checked separately
• Each failure maps to a pipeline stage

Sources

• Ragas metrics
• RAG survey