RAG or Fine-Tuning for Banks and Insurers: A Decision Guide

Your core banking vendor, your cloud provider, and three startups have all pitched you an LLM this year, and every pitch eventually lands on the same question: do you retrieve documents into the prompt, or do you fine-tune a model on your own data? For most banking and insurance workloads the answer is retrieval first, fine-tuning second, and usually a combination of both. The reason has less to do with benchmark scores and more to do with audit trails, document churn, and what your regulator considers a model.

This guide walks through the decision the way a technology or risk leader at a bank or insurer actually has to make it: what each technique changes, what compliance demands, what each option costs with current numbers, and a checklist you can apply to your own use case.

Two different levers: knowledge versus behavior

RAG and fine-tuning solve different problems, and most bad decisions come from confusing them.

Retrieval-augmented generation keeps the model frozen and injects relevant document chunks into the prompt at query time. The model reads your credit policy, claims handbook, or product terms fresh on every request. Knowledge lives in an index you control, not in the weights.

Fine-tuning updates the model's weights on your examples. It changes how the model behaves: the format it answers in, the classifications it prefers, the terminology it uses, the structure of its output. It is a poor way to store facts, because facts baked into weights are hard to update, hard to attribute, and hard to delete.

A useful shorthand: if the answer must cite a document that changes, that is a knowledge problem, use RAG. If the model must consistently behave a certain way regardless of which document it reads, that is a behavior problem, consider fine-tuning.

Start with the regulator, not the benchmark

In most industries you can pick an LLM architecture by accuracy and cost. In banking and insurance the compliance frame narrows the options before benchmarks matter.

Model risk management treats your LLM as a model

The Federal Reserve and OCC's SR 11-7 guidance on model risk management defines a model broadly: a quantitative method that processes input data into quantitative estimates used for business decisions, risk measurement, or compliance. If your LLM triages claims, drafts credit memos, or screens customers, examiners will treat it as a model. That means documented development, independent validation, and ongoing governance.

This is where the architecture choice bites. With RAG, the evidence chain is inspectable: for any answer you can show exactly which document version was retrieved, what the prompt contained, and what the model returned. With fine-tuning, the knowledge is distributed across billions of weights. You cannot point to where a fact came from, and you cannot surgically remove it when a policy is withdrawn. Validation teams strongly prefer systems whose inputs and outputs they can replay.

The EU AI Act makes credit and insurance pricing high-risk

If you touch EU customers, Regulation (EU) 2024/1689 classifies two core financial AI uses as high-risk in Annex III: systems used to evaluate the creditworthiness of natural persons or establish their credit score (point 5(b), with fraud detection carved out), and systems used for risk assessment and pricing of natural persons in life and health insurance (point 5(c)). The regulation's main obligations apply from 2 August 2026, so these systems are now in scope.

High-risk classification triggers requirements for risk management, data governance, technical documentation, logging, human oversight, and accuracy testing. None of that forbids fine-tuning, but every requirement is easier to evidence when facts come from a versioned retrieval index rather than from opaque weights.

The compliance-first conclusion: default to RAG for anything that informs a decision about a customer, and add fine-tuning only where it changes behavior you can validate.

Where RAG wins in a bank or insurer

RAG is the right primary pattern for most workloads teams bring to us:

• Policy and procedure Q&A. Credit manuals, underwriting guidelines, AML procedures, HR policies. Staff ask questions, the system answers with citations to the current paragraph. When the policy changes, you re-index a document, not retrain a model.
• Claims guidelines and coverage lookup. Adjusters query policy wordings and internal claims philosophies. Coverage terms change by product vintage, which is exactly the churn fine-tuning handles badly.
• Product and rate information. Anything with numbers that update weekly or daily must be retrieved, never memorized.
• Regulatory research. Answers that quote the regulation, circular, or directive clause they rely on.
• KYC and onboarding support. Retrieving the right checklist for a customer type and jurisdiction, with sources a compliance officer can verify.

Two properties make RAG dominant here. First, attribution: every claim links to a source document, which is what validators, auditors, and the hallucination controls you need in production are built around. Second, freshness: index updates are an operational task measured in minutes, not a training project measured in weeks.

RAG's honest limits: it will not teach the model your house style, it does not fix weak classification or extraction behavior, retrieval quality degrades on badly scanned or poorly structured documents, and long retrieved contexts raise per-query cost.

Where fine-tuning earns its keep

Fine-tuning pays when the problem is how the model responds, not what it knows.

• Structured extraction. Pulling fields from claims forms, invoices, or KYC documents into a strict schema. A tuned model holds format and edge-case handling far more consistently than prompting.
• Classification and routing. Triaging inbound claims by type, severity, and complexity; tagging complaints by regulatory category; routing servicing requests. These are behavior problems with stable labels, ideal for fine-tuning.
• House style and tone. Drafting customer letters, decline explanations, or advisor notes in your institution's voice with the required disclosures in the required places.
• Domain terminology. Making the model reliably use your internal product names, codes, and definitions.

The cost of fine-tuning collapsed with parameter-efficient methods. The QLoRA paper showed a 65B-parameter model can be fine-tuned on a single 48GB GPU by backpropagating through a 4-bit quantized model into low-rank adapters, with no measurable quality loss versus full tuning. That matters for budgets: the same paper notes full 16-bit tuning of a 65B model needs over 780GB of GPU memory, an 8-GPU node at minimum, while QLoRA fits on one card. We cover the operational side in our LLM fine-tuning best practices guide.

One hard constraint for regulated buyers: you cannot fine-tune Claude, and proprietary fine-tuning APIs mean your training data leaves your perimeter. Private fine-tuning therefore means open-weight models: Llama, Qwen, Mistral, or Gemma, trained inside your VPC.

The pattern that works: fine-tune the behavior, retrieve the facts

The strongest evidence that this is not either-or comes from Microsoft's RAG versus fine-tuning study, which built both pipelines on the same domain dataset and measured them on Llama 2, GPT-3.5, and GPT-4. Fine-tuning alone raised accuracy by over 6 percentage points. Adding RAG on top of the fine-tuned model added another 5 points. The gains stacked, because each technique fixed a different failure mode. In one experiment the fine-tuned model also generalized knowledge across regions, lifting answer similarity from 47% to 72%, which shows tuning can teach the model how to reason with a kind of data, not just memorize it.

Translated into a banking stack:

• The base model is an open-weight model hosted in your VPC, for example Llama deployed with private subnets, VPC endpoints, and customer-managed encryption keys, so inference traffic never touches the public internet.
• The fine-tuned adapter encodes behavior: extraction schema, triage labels, letter style, your terminology. It is retrained quarterly or when the behavior spec changes.
• The retrieval index holds facts: policies, rates, guidelines, regulation. It is re-indexed whenever a document changes, and every answer carries citations.

Each layer has its own validation story, its own change cadence, and its own owner. That separation is exactly what model risk teams ask for.

Worked example: a claims triage copilot, costed

Take a mid-size insurer building a copilot that reads inbound claims, classifies them, drafts an acknowledgement letter, and answers adjuster questions against the claims handbook. Here is the build with current, checkable prices.

Build phase: fine-tuning the behavior layer

• Training data: roughly 2,000 labeled historical claims for classification and extraction, plus a few hundred exemplar letters. The Microsoft study and QLoRA results both show small, high-quality datasets beat large noisy ones.
• Training compute: QLoRA tuning of a 70B-class open model on a single A100 80GB. On-demand, that card costs $2.79 per GPU hour at Lambda. A 24-hour run is about $67. Even ten experiment runs with evaluation cycles stay under $700 of compute.
• People cost dominates: expect most of the budget in data preparation and evaluation design, not GPUs.

Build phase: retrieval layer

• Index: claims handbook, policy wordings, letter templates, regulatory references, chunked and embedded.
• Vector store options: Pinecone starts free, then $50 per month minimum on the usage plan and $500 per month minimum on Enterprise, with a bring-your-own-cloud option that runs inside your VPC over PrivateLink. If your data must never leave your Postgres estate, pgvector self-hosted costs only the database you already run.

Run phase: monthly serving

• Volume assumption: 3,000 claims per month, roughly 50,000 tokens in and out per claim across classification, letter drafting, and adjuster Q&A, so about 150 million tokens.
• Serverless open-model inference as a marker: Together AI lists Llama 3.3 70B at $1.04 per million tokens, which puts this workload near $160 per month at API prices. Self-hosting on a dedicated GPU trades that per-token bill for a fixed instance cost plus operations time.
• Vector database: $50 to $500 per month depending on plan and residency requirements.

The point of the arithmetic is not the totals, it is where the money sits. Retrieval infrastructure is a rounding error next to document operations. Fine-tuning compute is a rounding error next to preparing the training data. The expensive parts are data curation, evaluation, and validation documentation, which is why we scope enterprise AI work for banks and insurers around governance deliverables, not GPU hours.

The decision checklist

Run every candidate workload through these questions, in order:

1. Does the answer depend on documents that change? If yes, you need RAG regardless of anything else. Fine-tuning cannot keep pace with document churn.
2. Must you show an examiner why the system said what it said? If yes, retrieval with citations is your evidence chain. Weights are not exhibits.
3. Is the failure you see about format, labels, tone, or terminology rather than missing facts? If yes, that is the fine-tuning candidate.
4. Is the use in an EU AI Act Annex III category, like creditworthiness or life and health insurance pricing? If yes, design for logging, human oversight, and technical documentation from day one, and prefer architectures whose inputs you can replay.
5. Do you have a few hundred to a few thousand high-quality labeled examples? If no, fix the data first. No training method compensates for labels your own experts disagree on.
6. Can training data leave your perimeter? If no, your fine-tuning shortlist is open-weight models in your VPC, full stop.
7. What changes more often: the facts or the behavior spec? Put the faster-changing half in the retrieval index.

If questions 1 or 2 say yes and question 3 says no, you are done: RAG only. If 3 says yes, add a fine-tuned behavior layer. Most serious deployments end up with both.

Common mistakes

• Fine-tuning on policy documents to teach the model the policy. The model memorizes a snapshot, then confidently cites last year's version after the update. Facts belong in the index.
• Buying the biggest model instead of evaluating. A well-tuned smaller model plus good retrieval routinely beats a frontier model with a weak index on domain tasks. Evaluate on your own labeled set before scaling up.
• No held-out test set. If you cannot measure accuracy before and after a change, you cannot validate the model or satisfy SR 11-7 style review. Build the eval set before the system, not after the incident.
• Treating the vector index as uncontrolled infrastructure. The index is part of the model system: version it, log what was retrieved, and include it in change management.
• Assuming private deployment means secure deployment. A VPC isolates network traffic, but you still need encryption with customer-managed keys, audit logging of inputs and outputs, and access control, as Meta's own private cloud deployment guide spells out.
• Skipping the retraining plan. Behavior specs drift as products and regulations change. Budget for periodic re-tuning and re-validation, or the system calcifies.

Frequently asked questions

Can we fine-tune Claude or GPT-4 on our banking data? Claude is not available for customer fine-tuning at all. Some proprietary providers offer fine-tuning APIs, but your training data is processed on their infrastructure, which most bank security policies rule out. Private fine-tuning in practice means open-weight models such as Llama, Qwen, Mistral, or Gemma trained inside your VPC.

Is RAG alone enough for regulatory compliance? It is a strong foundation, not a complete answer. Retrieval gives you citations, versioned sources, and replayable inputs, which map well onto model risk and EU AI Act logging expectations. You still need human oversight, documented validation, access controls, and an evaluation harness around it.

How much data do we need to fine-tune? For behavior tuning with methods like QLoRA, a few hundred to a few thousand high-quality examples is usually enough, and the research consistently shows small clean datasets outperform large noisy ones. The bottleneck is expert labeling time, not volume.

How often do we retrain versus re-index? Re-index whenever a source document changes, typically daily or on publication. Retrain the fine-tuned adapter when the behavior spec changes or evaluation scores drift, typically quarterly. If you find yourself retraining to keep facts current, that fact belonged in the index.

What does a minimal compliant RAG stack cost to run? A self-hosted stack on your existing Postgres plus a single inference GPU can run a departmental workload for a few hundred dollars a month in infrastructure. Managed vector databases start around $50 per month with enterprise tiers at $500 minimum. The dominant cost is always integration, evaluation, and validation effort.

Does this apply to fund operations and research teams too? Yes, the same knowledge-versus-behavior split holds for fund administration, reconciliation, and research workflows, which is how we structure our Claude for financial services implementations as well. The regulatory specifics differ, the architecture logic does not.

Where to go from here

The short version: retrieve facts, tune behavior, and let the compliance requirements pick the architecture before the benchmarks do. Most banks and insurers land on a RAG core with a small fine-tuned behavior layer, all running inside their own cloud perimeter. If you want that designed, deployed, and documented for your regulators, our enterprise AI service for banks and insurers covers private LLM deployment, fine-tuning, RAG, evaluation, and the governance documentation that goes with it.

Related reading

• SR 26-2 and GenAI Model Risk: What Banks Must Build Now