How to Validate an LLM Application Your Bank Did Not Build
US supervisors withdrew generative AI from model risk guidance in April 2026. Here is the practical validation playbook for a vendor LLM application your bank cannot inspect, from evaluation sets to monitoring and change control.

On April 17, 2026, the Federal Reserve, OCC, and FDIC replaced SR 11-7 with revised model risk guidance and explicitly took generative and agentic AI out of its scope. If you run model risk at a bank, that sentence has probably been quoted at you by someone who wants to ship an AI feature faster. Here is the uncomfortable part: the exclusion changes which rulebook applies, not whether your LLM application can hurt someone. This post is a practical playbook for the hardest version of the problem, validating a vendor LLM application whose weights, training data, and internals you will never see.
What actually changed in April 2026
The OCC's Bulletin 2026-13 did four things. It rescinded OCC 2011-12 (the OCC's twin of SR 11-7) along with older credit scoring and BSA/AML model issuances. It restated model risk management as a risk-based, principles-driven practice. It said the guidance is most relevant to banking organizations above $30 billion in assets and sets no enforceable standards. And in one carefully worded passage, it said generative and agentic AI models are "novel and rapidly evolving" and therefore not within scope.
Read the rest of that passage before celebrating. The same bulletin says the agencies plan a request for information on banks' use of AI, including generative and agentic systems. Supervisory action can still follow from unsafe or unsound practices caused by poor model risk management; the exclusion removes a checklist, not the expectation of safety and soundness. The pattern is familiar: regulators let industry practice mature and then codify it. That is exactly how SR 11-7 itself came to exist.
The industry signal points the same way. As Risk.net reported, most banks had avoided validating tools like Microsoft Copilot by classifying them as productivity software rather than models. But at least two large banks, Bank of America and Goldman Sachs, were validating generative AI applications by default before the old guidance was withdrawn. And Krishan Kumar Sharma, a model risk leader at Citi, published a six-pillar governance framework arguing that the GenAI system and each application built on it should be formally classified as a model and validated before deployment. When the banks with the most to lose keep validating after the requirement is lifted, that tells you where supervisory expectations are heading. We covered the regulatory shift itself in SR 26-2 and GenAI model risk: what banks must build now; this post is about the validation work.
First, decide what you are actually validating
The most common validation failure with LLM applications is validating the wrong object. Teams try to "validate GPT" or "validate Llama" as if the foundation model were the model. It is not. What your bank operates is an application: a base model at a specific version, a system prompt, a retrieval corpus, a set of tools, guardrails, and the evaluation set you test it against. Change any one of those and you have, in effect, a different model.
A useful rule of thumb: if a change could alter what a user sees for the same input, govern it as a model change. That means a vendor version bump, a prompt edit, a corpus refresh, or a guardrail update each trigger re-testing before release.
So the first validation artifact is a component inventory:
- Base model: provider, exact version or snapshot, hosting arrangement, and how version changes are communicated to you.
- Prompts: system prompt and any templates, stored in version control, not in someone's notebook.
- Retrieval corpus: the document set, its owners, its update cadence, and how entitlements survive retrieval.
- Tools and actions: what the application can do, not just say, with limits on each action.
- Guardrails: input filters, output filters, PII redaction, refusal rules.
- Evaluation set: versioned, with coverage by task and risk.
This inventory is also what your documentation hangs off; we wrote a separate working guide to LLM model risk documentation for banks if you need the document templates.
The three pillars, translated for a system you cannot inspect
SR 11-7 rested on three pillars: conceptual soundness, ongoing monitoring with outcomes analysis, and governance with independent effective challenge. The original guidance stated plainly that all model components should be subject to validation, and that this "applies equally to models developed in-house and to those purchased from or developed by vendors." The revised guidance keeps the same position on vendor products: you may never receive the code, data, or methodology, and the principles still apply. You cannot inspect the math of an LLM, so each pillar gets a behavioral translation, as practitioners like alt.qa have mapped it:
- Conceptual soundness becomes behavioral evidence. Instead of defending coefficients, you demonstrate on a versioned golden dataset that the application does what its intended-use statement claims, across the range of inputs it will actually see, including adversarial ones. Your eval coverage and results are the soundness argument.
- Ongoing monitoring becomes production sampling. You sample live outputs on a schedule, score them for accuracy, groundedness, and policy adherence, and track the scores as a time series. This matters more for hosted models than for anything SR 11-7 imagined, because the vendor can change the model underneath you without notice. Silent drift is a when, not an if.
- Effective challenge becomes reproducibility. A validator independent of the build team must be able to re-run your evaluation and get your numbers. That means versioned datasets, pinned model versions, hashed prompts, dated runs, archived results. An eval that lives in one engineer's notebook cannot survive challenge.
A six-step validation process for a model you did not train
This sequence adapts the classic vendor-model validation steps to an LLM application. It works for a vendor product, an API-based build, or an open-weight model you host yourself.
1. Fix the intended use, prohibited uses, and tier
Write down which decisions the application informs, who sees its output, and what it must never be used for. Drafting internal meeting summaries is a different risk from suggesting credit limits or answering customer complaints. Assign a materiality tier based on exposure and who acts on the output; this drives how much validation rigor is proportionate, which the revised guidance explicitly endorses.
2. Collect vendor evidence, and record the gaps
SR 11-7's vendor section reads as if it were written for LLM procurement. Require the vendor to provide developmental evidence explaining components, design, and intended use. Ask for their testing results showing the product works as expected. Ask them to state the model's limitations and assumptions and where use may be problematic. For a foundation model provider this means model cards, system cards, published evaluations, and safety documentation. Then do the part teams skip: write down what the vendor would not or could not give you, and how you will compensate for each gap. The revised guidance keeps the same vendor-product expectations SR 11-7 had: withholding code or data does not remove the obligation.
3. Review the retrieval and prompt design separately
Many wrong answers start in retrieval, not in the base model. Test the retrieval layer on its own: does it return the right passages for the questions your users ask, and do access controls survive retrieval so a clerk cannot retrieve a board paper? Then review the prompt: does it tie answers to sources, instruct refusals for out-of-scope questions, and prohibit the advice you must not give? We have a detailed procedure in how to evaluate a RAG system before a bank or insurer deploys it.
4. Run outcomes analysis on a versioned evaluation set
Because you cannot inspect internals, you lean on the two tools SR 11-7 names for exactly this situation: sensitivity analysis and benchmarking. In LLM terms, that is a golden evaluation set with known-good answers, scored against thresholds you agreed before you saw the results. The next section covers construction and numbers. Include adversarial prompts and, critically, questions your corpus cannot answer, so you can measure whether the application refuses instead of inventing. Document any customization you made, prompts, corpus choices, thresholds, and justify each one, because that documentation is itself a supervisory expectation.
5. Sample human review in production
Agree a review sample per period, stratified by task, customer segment, and language. Have reviewers record error categories, not a single pass rate: a 95% score that hides a fair-lending phrasing problem is worse than an honest 88% with a categorized tail. For high-impact outputs, keep a human in the loop before anything reaches a customer. Sharma, the Citi model risk leader cited above, calls human review the single most important control for hallucination risk, precisely because GenAI validation can never be as exhaustive as validation of a deterministic model.
6. Set monitoring metrics and change control
Monitor input mix drift, refusal rate, retrieval hit rate, judge scores on sampled outputs, and how often human reviewers overturn the application. The logging and evidence side of this is covered in LLM monitoring and audit logging for banks and insurers. Route every component change from your inventory through re-testing before release. And take seriously the contingency requirement from the vendor guidance: know how you would operate if the vendor deprecated the model version you validated, or the vendor itself. For API dependencies, that usually means a tested fallback model or a graceful degradation to human-only handling.
Building the evaluation set: a worked example
Abstract advice about golden datasets helps nobody. Here is a concrete build for a realistic case: a mid-size bank deploying a vendor RAG assistant that answers staff questions about internal lending policy.
Start with 400 cases, sized to the risk tier, and split them deliberately:
- 200 normal cases: real staff questions from pilot logs and the policy team's inbox, with known-good answers written by compliance subject matter experts.
- 80 edge cases: questions that span two policies, recently amended clauses, expired versus current versions, and ambiguous phrasing.
- 60 adversarial cases: prompt injection attempts, requests to ignore instructions, and users trying to get the assistant to commit the bank to terms. Our layered guide to prompt injection defenses for banks and insurers has the attack categories to draw from.
- 40 unanswerable cases: questions the corpus genuinely cannot answer. The only passing behavior is a refusal or escalation.
- 20 fairness probes: the same credit question phrased with different names, locations, and phrasing associated with protected characteristics, checking for disparate tone or content. If the application ever touches credit decisions, disparate-impact testing stops being a probe and becomes a release gate.
The finance-specific failure modes your set must target, beyond generic accuracy, are calculation errors (an assistant that confidently miscomputes an installment is worse than useless), hallucinated policy terms, missing mandatory disclosures, and inappropriate refusal or failure to refuse.
Scoring: use code checks for anything deterministic, numeric accuracy, required disclosure strings, schema validity. Use an LLM judge for semantic rubrics like groundedness and tone, but only after calibrating it: have your compliance experts label a sample, measure agreement between the judge and the humans, fix the rubric where they diverge, and version the judge alongside the dataset. A judge that was never calibrated against your own experts will not survive effective challenge, and should not.
On effort, a planning estimate rather than a market quote: for this assistant, expect the validator to spend five to six weeks end to end, compliance experts roughly 40 hours writing gold answers and calibrating the judge, and a developer about ten days on harness and instrumentation. Each subsequent vendor version change costs a two to four day re-run against the same versioned set. The expensive part is the first build; the marginal cost of staying validated is small, which is the entire point. If you want the harness details, see how to build an LLM evaluation suite for a bank or insurer.
What the validation report should contain
The report is what effective challenge actually reads. For an LLM application, it should have at least these sections:
- Intended use, prohibited uses, and tier, with the materiality reasoning.
- Component inventory from the register above, with versions and hashes.
- Vendor evidence and gaps: what you received, what was withheld, and how each gap is compensated.
- Design review findings for retrieval, prompts, and guardrails.
- Evaluation results against pre-agreed thresholds, broken down by case category, with error taxonomy.
- Judge calibration evidence: human agreement sample, rubric versions.
- Monitoring plan: metrics, sampling rates, alert thresholds, owners.
- Change control policy: which changes trigger full re-validation versus a targeted re-run.
- Limitations and residual risk: what the validation did not cover, stated plainly.
Where validation programs go wrong
- Validating the base model instead of the application. The vendor's published benchmarks are their evidence, not yours. Your exposure lives in your prompts, your corpus, and your users.
- Reporting one pass rate. Aggregate scores hide the failure categories that matter. A validator wants the error taxonomy and the tail.
- Letting the build team write all the test cases. That is developers grading their own homework. Independent challenge starts at dataset construction, not at the report.
- No re-test trigger for vendor version changes. If your validation predates the current model version, you do not have validation; you have a historical document.
- An uncalibrated judge. An LLM judge whose agreement with your own experts was never measured transfers trust, it does not create it.
- Treating the productivity-tool classification as permanent. Classifying a drafting aid as out of scope can be defensible. Never revisiting that decision as usage drifts toward customer-facing and decision-support work is not.
When a lighter touch is genuinely fine
The revised guidance is explicit that practice should be risk-based, and proportionality is real. A low-materiality internal drafting aid, where a competent human reviews everything before it is used, no customer sees raw output, and no decision is automated, does not need a 400-case evaluation set. What it needs is a written classification decision with the reasoning, basic usage guardrails, and a scheduled re-review, annually is a reasonable default, plus whenever the use case changes. The PRA's SS1/23 takes the same line in the UK: five principles, applied proportionately, with named senior accountability. The regime punishes the absence of a documented decision, not the existence of a small model tier. For the broader risk taxonomy that sits behind these choices, the NIST AI Risk Management Framework and its 2024 Generative AI Profile are the most useful neutral references to cite in your internal standard.
Frequently asked questions
Does SR 26-2 mean we no longer have to validate AI systems?
No. The April 2026 guidance excludes generative and agentic AI from its scope, but it sets no enforceable standards for anything and leaves controls for excluded systems to the bank's own risk management. Supervisors can still act on unsafe or unsound practices, and the agencies have signalled AI-specific guidance is coming through a planned request for information.
Can you validate a foundation model you cannot inspect?
You validate the application, not the weights. Supervisory guidance since 2011 has accepted that vendor products may withhold code, data, and methodology, and directs banks to lean on vendor developmental evidence, benchmarking, sensitivity analysis, and outcomes analysis on their own data. That is exactly the toolkit an LLM evaluation set operationalizes.
How big should the evaluation set be?
Big enough to cover your failure modes with statistical meaning, small enough to maintain. A few hundred well-chosen cases beat thousands of near-duplicates. Size to the tier: a low-risk internal assistant might justify 150 to 300 cases; anything customer-facing or decision-supporting wants more, plus a larger ongoing production sample.
Will validators accept LLM-as-a-judge scoring?
Increasingly yes, with conditions. The judge must be a different model from the one being tested, calibrated against your own subject matter experts on a labeled sample, versioned, and supplemented with deterministic checks and human review for high-impact outputs. An uncalibrated judge grading its sibling model convinces no one.
Does this apply outside the US?
The specific letters are US issuances, but the pattern is global. The UK's PRA SS1/23 defines models broadly enough that LLM applications can fall inside it, and expects proportionate MRM with senior accountability. Banks in Kenya and East Africa supervised by the Central Bank of Kenya are not bound by SR letters, but the same validation logic is what correspondent banking partners, investors, and increasingly local supervisors expect to see. The EU AI Act adds separate obligations for high-risk credit uses from 2027.
Where to go from here
Pick your highest-materiality LLM application, register its components this week, and run the six steps against it; most banks find the evaluation set and the change-control triggers are the two missing pieces. If you want a working evaluation harness and validation evidence pack built with you rather than handed to you, that is a core part of our enterprise AI work for banks and insurers, and our Claude for financial services practice covers the implementation side. Either way, validate now, while it is still your choice of framework rather than your supervisor's.
About AI Agents Plus Editorial
AI automation expert and thought leader in business transformation through artificial intelligence.



