How to Evaluate a RAG System Before a Bank or Insurer Deploys It

A RAG system that answers ten hand-picked demo questions correctly has proven almost nothing. Before a bank or insurer lets it answer real questions from staff or customers, it needs the same evidence any other consequential model needs: a fixed test set built from your own documents, separate scores for retrieval and generation, release gates, and documentation an independent reviewer can sign off. This post is the evaluation playbook: how to build the golden dataset, which metrics actually localize failures, how to set thresholds, and how to package the results so model risk and compliance can approve the deployment.

A demo is not evidence

Banks and insurers already know how to approve models. In the United States, the Federal Reserve and OCC's SR 11-7 supervisory guidance on model risk management treats any quantitative method that turns inputs into estimates used for business decisions as a model, and expects independent validation, documentation, and ongoing monitoring for it. A RAG assistant that answers questions about credit policy, product terms, or claims procedures fits that description comfortably, and supervisors in most jurisdictions apply the same logic through existing model risk rules even where no AI-specific regulation exists yet.

The failure modes are also specific to RAG, which generic chatbot testing misses. Researchers at Deakin University catalogued seven failure points from three RAG case studies:

• FP1, missing content. The question has no answer in the corpus. The system should abstain, but often guesses.
• FP2, missed the top-ranked documents. The answer exists but ranks below the top-K cutoff and never reaches the model.
• FP3, not in context. The right documents were retrieved but dropped when results were consolidated into the prompt.
• FP4, not extracted. The answer sits in the context but the model misses it, usually because of noise or contradictions.
• FP5, wrong format. The model ignores the requested output structure.
• FP6, incorrect specificity. The answer is too vague or too specific for the actual need.
• FP7, incomplete. The answer is not wrong, but omits information that was available.

Two conclusions from that paper should shape your evaluation plan: validation of a RAG system is only feasible during operation, and robustness evolves rather than being designed in at the start. A one-off pre-launch test is necessary but nowhere near sufficient. And a demo, one happy-path pass over a handful of curated questions, tells you nothing about any of the seven failure points.

Score retrieval and generation separately

A RAG system is two systems in series. The retriever turns a question into a ranked set of text chunks. The generator turns those chunks into an answer. Each can fail independently, and the fixes are different, so each needs its own scores.

The most widely used open framework is RAGAS, which evaluates both halves without requiring human-written reference answers for every case. Its documented metrics split cleanly:

• Retrieval side: context precision (are the top-ranked chunks actually relevant, in a useful order) and context recall (did the retrieved set contain everything needed to answer).
• Generation side: faithfulness (is every claim in the answer supported by the retrieved context) and answer relevancy (does the answer address the question asked).

For retrieval, also keep the classic information retrieval metrics. Recall@K, the share of questions whose known source chunk appears in the top K results, is the single most diagnostic number in RAG evaluation because it maps directly onto failure points FP2 and FP3.

The reason for the split becomes obvious the first time an answer fails. If faithfulness is low, either the retriever supplied the wrong context (a retrieval problem) or the model added claims the context does not support (a generation problem). With only an end-to-end score you cannot tell which, and teams routinely burn weeks tuning prompts when the real problem is chunking or ranking. If you are still choosing the retrieval stack itself, our comparison of vector databases for bank RAG covers that decision; the evaluation method here works regardless of which store you pick.

Build a golden dataset from your own documents

Every metric above is only as good as the test set it runs against. The golden dataset is the fixed set of questions, with known correct sources and expected behavior, that every evaluation run and every release gate uses. Build it once, version it, and grow it.

1. Pick one bounded use case. Internal policy Q&A, claims handbook Q&A, product disclosure Q&A. Do not try to evaluate "the RAG system" in general; evaluate one deployment against its own corpus.
2. Collect 50 to 100 real questions. Pull them from search logs, helpdesk tickets, pilot users, and the people who currently answer these questions by email. Braintrust's RAG evaluation guidance suggests starting with 5 to 10 representative cases and growing from there; for a go or no-go decision at a bank you want at least 50, so each failure category has enough cases to mean something.
3. Have a subject-matter expert label each case. For every question, record the source document and section containing the answer, a short reference answer, and the expected behavior (answer, or abstain). This labeling is the most expensive step and the one you cannot skip, because Recall@K and faithfulness scoring both depend on knowing what correct looks like.
4. Cover all four case types. Successful query-answer pairs that must keep working; known past failures you have fixed; questions requiring synthesis across several documents; and questions that should return an insufficient-information answer because the corpus does not cover them. The last type is what tests FP1, and it is the one most teams forget.
5. Add near-miss traps. Questions whose keywords appear in the corpus but whose answer does not; questions about superseded policy versions; questions that mix two products with similar names. These separate a system that knows what it does not know from one that always says something.
6. Version and freeze. Each evaluation run records the dataset version. Grow the set monthly from confirmed production failures, never by quietly editing old cases.

Keep the dataset private. If test questions leak into training or fine-tuning data, your scores measure memorization, not performance.

The metrics that matter and how to read them

There is no universal passing score for any RAG metric. What matters is choosing thresholds deliberately, calibrating the scorer, and watching trends.

A few readings that hold up in practice:

• Recall@K is your retrieval health check. If Recall@5 is 0.78, roughly one question in five is unanswerable before the language model is even involved. Fix chunking, metadata, or ranking before touching prompts.
• Context precision tells you about noise. Relevant chunks buried under irrelevant ones cause FP4-style extraction failures. This is not hypothetical: the Lost in the Middle study showed that model performance degrades significantly when relevant information sits in the middle of a long context, even in models built for long contexts. Fewer, better-ranked chunks beat more chunks.
• Faithfulness is the compliance metric. It measures whether the answer sticks to the retrieved evidence. When it drops, inspect the failing traces: which context was available, and which claims were unsupported. That distinction tells you whether to fix retrieval or generation.
• Answer relevancy catches polite non-answers. A response can be faithful to the context and still not answer the question.

Calibrate the LLM judge before you trust it. Sample 30 to 50 scored cases, have humans re-grade them, and measure agreement. If the judge is systematically harsher or more lenient than your reviewers, fix the rubric before wiring scores into any release decision.

A worked example: evaluating an internal policy assistant

Here is what one full evaluation cycle looks like. The numbers are from a hypothetical but realistic run; they illustrate the process, not a benchmark.

Setup. A bank is deploying an assistant over 4,000 internal policy and product documents. Golden dataset: 120 labeled questions, split into 60 single-document lookups, 30 multi-document synthesis, 20 insufficient-information cases, and 10 previously observed failures.

Baseline (512-token chunks, embedding search, top 5, no reranking):

• Recall@5: 0.78
• Context precision: 0.71
• Faithfulness: 0.83
• Answer relevancy: 0.88
• Correct abstention on insufficient-info cases: 11 of 20

Failure analysis of the 20 lowest-scoring answers: 13 were retrieval failures (the gold chunk never reached the prompt), 4 were extraction failures where the answer sat mid-context, and 3 were confident answers to questions that should have been declined.

Changes. Add short chunk-specific context headers before embedding, add a cross-encoder reranker, and add explicit abstention instructions to the system prompt. The first change is not guesswork: Anthropic's Contextual Retrieval work found that prepending explanatory context to each chunk reduced failed retrievals by 49 percent, and by 67 percent when combined with reranking, with failure measured as the relevant chunk missing from the top 20.

Re-run on the same frozen dataset:

• Recall@5: 0.93
• Context precision: 0.86
• Faithfulness: 0.94
• Answer relevancy: 0.91
• Correct abstention: 18 of 20

Release gate. The team sets: Recall@5 at 0.90 or above, faithfulness at 0.93 or above, abstention accuracy at 90 percent or above, and zero unsupported-claim findings in a human audit of 30 sampled answers. The candidate passes; the baseline did not. Note that the gate combines automatic scores with a human audit. No metric alone carried the decision.

The same method transfers directly to financial-services deployments such as fund administration and research assistants, the audience our Claude for Financial Services work is built for.

Gate releases and monitor production

Evaluation earns its keep after launch, not before it. Three mechanisms:

• CI gates. Run the golden dataset on every change to chunking, embedding model, reranker, prompt, or generator model, and block the release when any gated metric regresses. Compare the candidate against the currently accepted version and repeat generations, because LLM output variance can masquerade as a regression on a single run.
• Production sampling. Score a sample of live traffic weekly with the same metrics, alert when scores drop below the accepted baseline, and route confirmed failures back into the golden dataset. This is the loop the seven-failure-points paper means by validation only being feasible during operation.
• Re-validation triggers. Define them in writing: corpus re-index, model version change, prompt change, or a drift alert. Any trigger re-runs the full gate.

The tooling cost is modest next to the engineering time. As of 2026, Confident AI's comparison of RAG evaluation tools lists its own tiers at a free plan, then 200 dollars a month for Starter and 2,000 dollars a month for Team, with Braintrust's Pro tier at 249 dollars a month; open-source RAGAS itself is free. Budget also for judge-model tokens per run and, the largest line, subject-matter expert hours for labeling and human audits.

Document it like a model validation

The evaluation only counts if someone independent can review it. SR 11-7's core expectation is validation performed by staff who were not responsible for development, documented thoroughly enough for a third party to understand and challenge the work. Package your RAG evaluation the same way:

• System description. Corpus scope and size, chunking strategy, embedding model, reranker, generator model and version, system prompt.
• Dataset documentation. Size, provenance, labeling process, coverage of the four case types, dataset version used for the run.
• Metrics and thresholds. Each metric, its threshold, and a one-line justification tied to the risk of the use case.
• Results and failure analysis. Per-metric scores, the failure breakdown, and known limitations, for example weak abstention on out-of-scope questions.
• Judge calibration. The human-versus-judge agreement study and its date.
• Monitoring plan. Sampling rate, alert thresholds, re-validation triggers, owners.
• Sign-offs. System owner and independent reviewer.

This pack is what turns "we tested it" into evidence an auditor or supervisor can work with. It slots into the broader model governance work covered in our guide to building an LLM evaluation suite for a bank or insurer, and it should be paired with adversarial testing, since a golden dataset measures expected behavior, not hostile input. Our red-teaming guide covers that side.

Common mistakes

• Scoring only the final answer. You lose the ability to localize failures between retrieval and generation.
• Letting the build team write all the test questions. The set drifts toward what the system already handles. Include questions from people who did not build it, and hold some back.
• No insufficient-information cases. Abstention never gets tested, and FP1 failures reach production.
• Trusting the LLM judge without calibration. Unexamined rubric drift makes scores meaningless over time.
• Treating evaluation as a launch gate only. Corpus changes, model upgrades, and new query patterns all invalidate old results.
• Optimizing averages. A system can average 0.93 faithfulness while failing every multi-document synthesis question. Report per category.

When this is the wrong approach

• Low-stakes internal tools with human review of every answer. A 30-question set and a weekly manual review may be all the assurance the use case justifies.
• The system is not really document Q&A. Structured extraction or transactional lookups should be tested as data pipelines with exact-match and schema checks, not with RAGAS-style semantic metrics.
• Tiny static corpora. If the whole corpus is one 40-page manual, careful prompt design plus human spot checks will deliver more than the overhead of a full evaluation harness.
• No access to subject-matter experts. If nobody can label correct answers, do not manufacture synthetic labels and call it validation. Narrow the launch scope until someone can.

Frequently asked questions

How many questions does a RAG golden dataset need? For an early development loop, 10 to 20 well-chosen cases catch most regressions. For a go or no-go decision at a bank or insurer, aim for at least 50 to 100, with each failure category (single-document, multi-document, insufficient-information, known past failures) holding enough cases for the score in that category to mean something.

What is a good faithfulness score for a bank RAG system? There is no universal number, because scores depend on the judge model and rubric. For high-stakes answers, teams commonly gate at 0.9 or above after calibrating against human grading, paired with an abstention test and a human audit sample. The trend line matters more than the absolute value.

Can we use Claude as the judge in RAGAS? Yes. RAGAS is model-agnostic: you configure which LLM scores each metric. Whatever model you choose, run the human calibration study described above before wiring scores into release gates, and pin the judge model version so results stay comparable across runs.

How often should we re-run the evaluation? On every change to the corpus, chunking, embedding model, reranker, prompt, or generator model; weekly on sampled production traffic; and in full whenever a documented re-validation trigger fires. Annual full re-validation is a reasonable floor for a material deployment.

Is synthetic test data good enough? It is useful for coverage but risky as a foundation. Unreviewed synthetic questions tend to reflect the generator's habits, and scores against them flatter the system. Use synthetic data to expand volume, have humans review it, and treat confirmed production failures as the highest-value additions.

Where to go from here

If your institution is past the demo stage and needs a RAG deployment that model risk, compliance, and audit can sign off, the evaluation pack described here is part of how we deliver enterprise AI for banks and insurers: private deployment, evaluation harnesses, and the governance documentation to go with them.