Evaluating RAG Beyond a Good Demo

A retrieval-augmented generation feature can look impressive in a demo and still fail when real users depend on it. A fluent answer is not enough. The system needs to find the right source material, use it faithfully, and make uncertainty visible when the available context is weak.

I think about evaluation as a pipeline rather than one final score. Retrieval should be checked first: did the relevant document appear, how highly was it ranked, and did the query contain enough information to find it? Generation comes next: does the answer follow the retrieved context, include useful citations, and avoid inventing details that are not supported?

A small, carefully chosen test set is more useful than judging random outputs. It should include ordinary questions, ambiguous requests, missing information, conflicting documents, and cases where the correct response is to ask for clarification. Running the same set after changes to chunking, embeddings, prompts, or models makes regressions easier to see.

Human review remains important because usefulness is contextual. A technically grounded answer may still be unclear, too long, or poorly suited to the workflow. Combining repeatable checks with reviewer feedback produces a better picture than either approach alone.

The goal is not to prove that an AI feature is perfect. It is to know where it is reliable, where it needs help, and whether each change makes the system meaningfully better.