A RAG system without evaluation is a guess: a 2026 practical guide to measuring retrieval and generation separately

This is the third piece in the production RAG cluster, completing a diagnosis → chunking → evaluation loop. RAG did not solve hallucinations named the failure modes. Every 512 tokens is not a chunking strategy fixed the number-one failure mode. This piece closes the loop with the discipline that tells you whether either of those fixes actually worked: evaluation.

My read, after going through the RAG evaluation literature, is blunt: a RAG system without evaluation is not a system you are operating — it is a system you are guessing about. The guess feels reasonable because the outputs look plausible. But "plausible" is the most dangerous word in production RAG, because the same fluency that makes an answer look right also makes a wrong answer look right. The only way to tell the difference is to measure, and the only way to measure usefully is to split the metrics across the two layers that can independently fail: retrieval and generation.

The core principle: measure retrieval and generation separately

This is the single most important idea in RAG evaluation, and it is the connective tissue across this whole cluster. Recall the data point from the diagnosis piece: when RAG fails, roughly 73% of the time the failure is in retrieval. If you measure only the final answer, you cannot tell whether a bad answer came from bad retrieval (the model faithfully reported wrong context) or bad generation (the model ignored or went beyond good context). Those two failure modes require completely different fixes, and a single end-to-end metric cannot distinguish them.

The discipline this implies: evaluate the retrieval layer and the generation layer as separate things, with separate metrics, every time. Retrieval metrics tell you whether the right context came back. Generation metrics tell you whether the model used that context correctly. Together, they localize the failure; apart, they leave you guessing.

The four metrics that actually matter

The canonical 2026 RAG metric set — pioneered by Ragas and adopted across DeepEval, Phoenix, and the broader ecosystem — is four metrics, split two-and-two across the layers.

Retrieval-layer metrics (did we fetch the right context?)

• Context precision. Of the chunks retrieved, how many were actually relevant to the query? High precision means the retriever is not polluting the context window with irrelevant noise. Low precision means the model has to extract signal from garbage.
• Context recall. Of the relevant chunks that existed in the corpus, how many did retrieval actually find? High recall means the retriever is not missing the chunk that contained the answer. Low recall means the model never had a chance — the right context was in your index, but the retriever did not surface it.

These two are in tension, like any precision/recall pair. Tightening retrieval to boost precision can lower recall (you fetch less, and miss some relevant chunks). Loosening it to boost recall can lower precision (you fetch more, including irrelevant noise). The right balance depends on your downstream tolerance for noise, which is exactly what the generation metrics measure.

Generation-layer metrics (did the model use the context correctly?)

• Faithfulness. Is every claim in the answer supported by the retrieved context, or did the model go beyond it (the classic RAG hallucination)? This is the metric that catches the failure mode the diagnosis piece named "RAG did not eliminate hallucinations — it moved them." A faithful answer stays inside the context; an unfaithful one reaches outside it, even when the outside claim happens to be true.
• Answer relevance. Does the answer actually address the question that was asked? A model can be perfectly faithful to irrelevant context and still produce a useless answer. Answer relevance catches the "you answered something, but not what I asked" failure.

The discipline: track all four. A RAG system that is high on retrieval metrics but low on faithfulness has a generation problem (the model is going beyond context). A system that is high on faithfulness but low on context recall has a retrieval problem (the model is faithfully reporting incomplete context). The four metrics together tell you which layer to fix; any subset leaves you guessing.

The production threshold: faithfulness around 0.75

A useful concrete anchor from the practitioner literature: a faithfulness score around 0.75 or higher is a common production deployment threshold (cited in the DataVLab 2026 RAG evaluation guide). Below that, the model is making things up often enough that users will notice, and you should not ship to production without investigating why.

The threshold is not a law — your tolerance depends on your use case. A medical or legal RAG system should demand higher faithfulness than an internal doc-search tool. But 0.75 is a useful sanity check: if your faithfulness is below it, you have a problem worth solving before launch; if it is well above it, faithfulness is probably not your bottleneck and you should look at retrieval recall instead.

Offline evaluation vs. online evaluation

The 2026 RAG evaluation stack splits into two modes, and you need both.

Offline evaluation runs on a curated dataset before deployment. You build a set of query-context-answer triples (this is the golden set discipline from our eval cluster, applied to RAG), run your pipeline over it, and compute the four metrics. Offline eval is how you catch regressions before they hit users — when you change chunking, embedding, retrieval, or the prompt, you re-run offline eval and check whether the metrics moved the wrong way. Tools like Ragas, DeepEval, and Phoenix are the workhorses here.

Online evaluation runs in production, on real traffic, continuously. You sample real queries, score them (often with an LLM-as-a-judge, calibrated against human labels per our judge calibration piece), and track the four metrics over time. Online eval is how you catch the failures your offline set did not predict — the new query types, the edge cases, the drift. Platforms like LangSmith, Phoenix, and FutureAGI provide the infrastructure.

The split matters because the two modes catch different things. Offline catches regressions you can predict; online catches the ones you cannot. A team with only offline eval is blind to production reality; a team with only online eval is fixing problems after users see them. You need both.

How this connects to the rest of the cluster

This piece completes the RAG cluster's diagnostic loop and connects outward:

• It measures whether the chunking strategy you chose is actually working — context recall is the metric that tells you chunking is right.
• It localizes the failures the diagnosis piece named — the four metrics map directly onto the five failure modes in that piece.
• It applies the golden set discipline to RAG specifically — a RAG eval set is a golden set of query-context-answer triples.
• It relies on the judge calibration discipline for online evaluation — an uncalibrated judge scoring faithfulness in production is decorative, same as anywhere else.
• It pairs with per-task cost observability — quality (from RAG eval) + cost (from observability) is the full decision matrix for retrieval configuration choices.

The sharp edges that are not in the marketing copy

A few risks worth knowing:

• Your RAG eval set is itself a golden set, and it rots. The same refresh discipline applies: sample real production queries, add edge cases, version the set. A RAG eval set frozen at launch stops predicting reality within months.
• Faithfulness does not catch factually-wrong-but-context-supported answers. If your retrieved context is itself wrong (stale source, indexing error), the model can faithfully report it and score high on faithfulness while being factually wrong. Faithfulness measures grounding, not ground truth.
• The four metrics do not cover everything. They do not measure latency, cost, safety, or PII leakage — all of which matter in production. Treat the four as the quality core, not the complete picture.
• LLM-judged RAG metrics inherit the judge's biases. Position bias, verbosity bias, self-preference — all apply when an LLM scores faithfulness. Calibrate the judge, or your faithfulness score is a vibe.
• Aggregate metrics hide distributional failures. A system with 0.85 average faithfulness can still be catastrophically bad on a specific query category that averages out. Slice the metrics by query type, not just in aggregate.

How to actually build this in 2026

The practical path I would give a team:

1. Build a RAG eval set first. Fifty to a few hundred query-context-answer triples, drawn from real production queries plus crafted edge cases. This is the asset everything else depends on.
2. Run the four metrics offline on your current pipeline. Use Ragas or DeepEval. Establish a baseline before changing anything.
3. Set a faithfulness threshold for production (around 0.75 to start). Do not ship below it without a documented reason.
4. Add online evaluation on a sample of real traffic. Calibrate the judge against human labels. Track the four metrics over time.
5. Use the metrics to localize every regression. When quality drops, the four-metric split tells you whether to fix retrieval or generation. Never debug from the final answer alone.
6. Slice the metrics by query type. Aggregate averages hide the categories that cause incidents.
7. Refresh the eval set from production on a cadence. A RAG eval set that does not evolve stops predicting reality.

My take

The 2026 story is that RAG evaluation is the discipline that separates a system you operate from a system you guess about. The four metrics — context precision, context recall, faithfulness, answer relevance — are not exotic; they are the minimum viable instrumentation that tells you whether your retrieval and generation layers are actually working, independently of each other. A team that measures all four, on a real eval set, with a calibrated judge, online and offline, can see exactly where RAG fails and fix the right layer. A team that measures only the final answer is debugging in the dark.

If you take one thing from this cluster: RAG failure is layered, and RAG evaluation must be layered to match. Measure retrieval and generation separately, every time, or you are guessing about which layer to fix.

This is the third and final piece in the production RAG cluster. Start with RAG did not solve hallucinations — it moved them for the failure diagnosis, then Every 512 tokens is not a chunking strategy for fixing retrieval's number-one failure mode, then this piece for the evaluation that tells you whether any of it worked. For the broader evaluation discipline these metrics depend on, see the LLM evaluation cluster. For a maintained provider reference, see our AI pricing data page.

Sources

• Ragas official docs: metrics (faithfulness, answer relevancy, context precision/recall)
• Ragas docs: Faithfulness metric
• DataVLab: RAG evaluation 2026 — methods, metrics, frameworks (faithfulness threshold ~0.75)
• FutureAGI: Top 5 tools to evaluate RAG performance in 2026
• Braintrust: Best RAG evaluation tools in 2026, compared
• QASkills: RAG evaluation metrics 2026 — the complete guide
• Comet: RAG evaluation guide — metrics, methods, key quality signals
• Adaline: Best RAG evaluation tools in 2026
• arXiv 2504.14891: Retrieval-augmented generation evaluation in the era of LLMs (survey)
• SapotaCorp: RAG evaluation with Ragas — faithfulness, context recall, relevance
• Our RAG cluster: RAG did not solve hallucinations — it moved them
• Our RAG cluster: Every 512 tokens is not a chunking strategy
• Our eval cluster: golden set construction
• Our eval cluster: LLM-as-judge calibration
• Our pricing cluster: per-task cost observability