Teams build a RAG system, it "mostly works," and then they're stuck — because they can't tell why it fails or whether a change helped. The fix is evaluation, and RAG needs it in two distinct places.
Two stages, two failure modes
RAG has two parts, and each can break independently:
- Retrieval — did it fetch the right context? If the relevant document wasn't retrieved, even a perfect model can't answer.
- Generation — given the retrieved context, did the model produce a faithful, correct answer? Or did it ignore the context, or hallucinate beyond it?
Debugging without separating these is guesswork. A wrong answer might be a retrieval miss or a generation failure — and the fixes are completely different.
Don't ask "is the answer good?" Ask "did we retrieve the right thing?" and "did the model use it faithfully?" — two questions, two fixes.
Retrieval metrics
Measure whether the right documents show up: did the relevant chunk appear in the top results, and how highly was it ranked? Build a set of questions with known-correct source documents, and check how often retrieval surfaces them. Low retrieval scores mean fix your chunking, embeddings, or add re-ranking — not your prompt.
Generation metrics
For the answer, measure faithfulness (does it stick to the retrieved context, or make things up?) and relevance (does it actually answer the question?). An LLM-as-judge, given the question, context, and answer, can score these — validated against human ratings.
The workflow
Build an evaluation set from real questions with expected sources and answers. Run it whenever you change anything — chunking, model, prompt — and watch both retrieval and generation scores. This turns "it feels better" into "retrieval recall went up 12%," which is how you actually improve a RAG system instead of guessing.
