Private LLM GPU Sizing for Banks: Hardware and Serving Stack Costs

A bank that wants a private LLM faces two buying decisions, not one: the GPUs, and the serving software that turns those GPUs into an API your applications can call. Get the hardware wrong and you either leave half your spend idle or discover, after delivery, that your model does not fit in memory. Get the software wrong and a single H100 serves fewer users than a card a tenth its price should. This post gives you the memory arithmetic, a serving-stack decision, and current buy-versus-rent numbers, so you can size a deployment before you sign a purchase order.

Start with the workload, not the GPU

Every correct sizing starts from four numbers that have nothing to do with hardware:

• Peak concurrent sessions. Not headcount. A 2,000-employee bank might see 30 people querying the model at the same moment. A 200-person fund administrator running document review batches might hit 50 concurrent requests from automated pipelines alone.
• Context length per request. A KYC analyst pasting a 40-page document uses roughly 20,000 to 30,000 tokens of context. Long contexts multiply memory use per session, as you will see below.
• Tokens per second per user. Interactive chat feels acceptable at 30 to 50 tokens per second per user. Batch pipelines care about aggregate throughput instead.
• Model class. Open-weight models come in rough size classes: 7B to 14B parameters (fast, cheap, weaker reasoning), 30B to 40B (a reasonable middle), and 70B (the workhorse class for serious document work, such as Meta's Llama 3.1 70B at roughly 70 billion parameters).

Write these four numbers down. They drive everything else.

The memory math that decides what fits

GPU memory is the binding constraint in LLM serving, and the arithmetic is simple enough to do on an envelope.

Weights. Memory for model weights equals parameter count times bytes per parameter. At BF16 precision (2 bytes per parameter), a 70B model needs about 140 GB. That matches reality: the Llama 3.1 70B safetensors files are stored in BF16. An 8B model at BF16 needs about 16 GB.

Quantization. You can store weights at 4-bit precision instead of 16-bit, cutting memory roughly four times: a 70B model drops to about 35 to 40 GB including overhead. The method that made this practical is activation-aware weight quantization (AWQ), which won the MLSys 2024 best paper award. The AWQ paper shows that protecting just 1 percent of the most important weights removes most of the quality loss, and it needs no retraining. The Llama 3.1 70B model page alone lists over 120 community quantization variants.

KV cache. Every active session also needs memory for its attention state, and this grows with concurrent sessions multiplied by context length. This is the part first-time buyers forget. Weights for a 4-bit 70B model fit in 40 GB, but 30 analysts each carrying 25,000-token contexts can consume tens of gigabytes more. Plan 30 to 50 percent headroom beyond the weights.

What this means in practice:

• A 70B model at 4-bit fits on one 80 GB GPU (A100 or H100) with about 40 GB left for KV cache and traffic spikes.
• A 70B model at full BF16 quality needs two 80 GB GPUs with tensor parallelism splitting the weights.
• An 8B to 14B model, even at BF16, fits on a single 24 GB to 48 GB card.

For reference, the NVIDIA H100 ships with 80 GB of HBM3 memory and 3.35 TB per second of memory bandwidth, which is why it has become the default unit of account for this kind of sizing.

Three sizing tiers with real numbers

Tier 1: pilot or single team. One 8B to 14B model on a single 24 to 48 GB GPU. Serves 5 to 15 concurrent users comfortably. This is the right size for a two-month proof of concept in one department, and it is cheap enough to rent by the hour.

Tier 2: one serious workload. A 70B model at 4-bit on one 80 GB GPU, or two 48 GB GPUs. In one independent benchmark comparison of serving engines, a single A100 80 GB running Llama 3.1 70B at 4-bit with 16 concurrent sessions delivered 2,800 to 3,400 tokens per second in aggregate with the right serving engine. That is roughly 175 tokens per second per user at 16 concurrent sessions, far above reading speed, with room to grow toward 30 or 40 sessions as batching efficiency improves.

Tier 3: firm-wide platform. A 70B model at BF16 across two 80 GB GPUs, or several quantized replicas behind a load balancer for higher concurrency and redundancy. This is where eight-GPU nodes enter the conversation.

Three warnings about tier 3. First, a single replica is a single point of failure, and a bank that makes a model part of a regulated workflow needs N+1 capacity. Second, bigger is not automatically better: if your eval set shows an 8B or 32B model passing your quality bar on your documents, a tier 1 or 2 box will serve the whole organization at a fraction of the cost. Size to the eval, not to ambition. Third, a single node with one to eight GPUs covers everything through this tier; multi-node clusters with InfiniBand are training infrastructure, and you should not let a vendor sell you a training cluster for an inference workload.

The serving stack decision in 2026

The serving engine matters as much as the GPU. An empirical comparison published in late 2025 measured vLLM delivering up to 24 times the throughput of Hugging Face TGI under high concurrency on LLaMA-2 models, while TGI held a tail-latency edge for single interactive users. In practice the choice in 2026 is narrower than the blogosphere suggests:

vLLM is the default. The vLLM documentation lists the features that matter to a bank: continuous batching, PagedAttention memory management, prefix caching, FP8 and INT4 quantization support including AWQ, tensor and pipeline parallelism, and an OpenAI-compatible API server that also speaks the Anthropic Messages API. That API compatibility matters: applications and agent frameworks written against either vendor SDK can be pointed at your private endpoint with a configuration change. vLLM supports over 200 model architectures across NVIDIA, AMD, and CPU backends.

Do not start new builds on TGI. Hugging Face has archived the Text Generation Inference repository and states in the README that the project is in maintenance mode with only minor fixes, recommending vLLM, SGLang, and llama.cpp going forward. TGI was a solid choice in 2024. Recommending it for a new deployment in 2026 means building on software its own maintainer has stopped actively developing.

llama.cpp for small scale and edge. llama.cpp is MIT-licensed C++ with no Python dependency, runs on CPUs and almost any GPU, supports CPU+GPU hybrid inference for models larger than your VRAM, and ships an OpenAI-compatible server. Its throughput under concurrency trails vLLM badly (900 to 1,300 tokens per second versus 2,800 to 3,400 in the benchmark above), but for a pilot, a branch office, or a disaster-recovery fallback on cheap hardware, it is the right tool.

A production vLLM launch looks like this:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --quantization awq \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --api-key $INTERNAL_KEY \
  --port 8000

TensorRT-LLM and SGLang are legitimate options for teams with deep NVIDIA expertise chasing maximum tokens per dollar; NVIDIA cites SemiAnalysis benchmarks showing a B200 with TensorRT-LLM at about $0.02 per million tokens versus $0.09 on an H100 with vLLM. Treat those as second-generation optimizations, not day-one choices.

Cost breakdown: buy, rent, or call an API

Here are the numbers a CFO will ask for, with sources.

Renting. Lambda's published GPU cloud pricing, billed by the minute with no egress fees:

• One H100 80 GB: $4.29 per GPU-hour, about $3,130 per month running continuously.
• Two H100s: $8.38 per hour as a 2-GPU instance, about $6,115 per month continuous. This covers a tier 3 BF16 setup or two redundant tier 2 replicas.
• Eight H100s: $3.99 per GPU-hour, $31.92 per hour, about $23,300 per month continuous.
• Cheaper tiers: an A100 80 GB runs $2.79 per GPU-hour, and a 40 GB A100 runs $1.99 per GPU-hour, which suits pilots.

Buying. Integrated eight-GPU HGX H100 servers from OEMs such as Supermicro, Dell, and HPE quote at $250,000 to $320,000 in 2026, with a typical mid-range quote near $285,000. A European vendor's published price guide lists a Dell PowerEdge XE9680 with eight H100s at EUR 198,000 to 290,400 excluding VAT, which corroborates the range. A single H100 PCIe card lists around EUR 15,000 to 28,000.

Break-even arithmetic. Renting eight H100s continuously costs about $23,300 a month; buying the node costs roughly $285,000. Pure hardware break-even is about 12 months of fully utilized rental. Add power, cooling, colocation or datacenter space, spare parts, and the staff time to operate the box, and a realistic break-even lands closer to 15 to 18 months at high utilization. Below roughly 60 percent utilization, renting wins for a long time. This is arithmetic on published prices, not a quote; your numbers will differ.

Calling an API instead. Per-token pricing needs no capex and scales to zero at night. NVIDIA's own H100 page cites inference at about $0.09 per million tokens on an H100 with vLLM. If your total volume is a few hundred million tokens a month, self-hosting anything is hard to justify on cost alone; the case for private deployment at that scale is regulatory, not financial.

The costs nobody puts in the spreadsheet. Redundancy (a second replica), model re-validation when you upgrade, evaluation infrastructure, monitoring and audit logging, and the engineer-hours to patch the serving stack. Budget for them explicitly or the project stalls in month four.

A rollout procedure that survives an audit

1. Freeze the workload definition. Peak concurrent sessions, context length, tokens per second target, model class. Get the business owner to sign it.
2. Build the eval set first. Fifty to two hundred real tasks from your documents, scored. Every later decision (model size, quantization, engine) gets validated against this set; our guide to building an LLM evaluation suite for a bank or insurer walks through the construction, and RAG versus fine-tuning for banks and insurers covers how model choice and data strategy interact here.
3. Rent before you buy. Run one to two months on rented GPUs. Validate the eval scores, measure real throughput at your real context lengths, and confirm the concurrency target is right.
4. Size with the memory math. Weights plus KV cache plus 30 to 50 percent headroom, then add N+1 for redundancy.
5. Run the break-even. Utilization estimate against the rent-versus-buy numbers above. If you have not decided between VPC and on-premise at all yet, our comparison of private LLM deployment options for banks is the upstream decision.
6. Deploy behind your existing gateway. The OpenAI-compatible endpoint means your applications see a URL and an API key; nothing else changes.
7. Instrument from day one. Latency percentiles, tokens per second, GPU memory, and full request audit logs. Our post on LLM monitoring and audit logging for banks and insurers covers what supervisors expect to see.
8. Write the model risk file. Regulators increasingly ask for it; our working guide to LLM model risk documentation gives the structure.

Common mistakes

• Sizing for headcount instead of concurrency. A 5,000-employee institution rarely needs more than 40 concurrent sessions. Buying for headcount wastes six figures.
• Forgetting the KV cache. The model fit in memory during the demo with 2,000-token prompts, then ran out of memory in production with 30,000-token documents. Do the math at production context lengths.
• Starting on TGI because an older tutorial used it. The project is archived. Use vLLM.
• Running BF16 when 4-bit would pass. Full precision doubles your hardware bill for the 70B class. Test 4-bit on your eval set first; if scores hold, take the savings. Sometimes they do not hold, and that is what the eval set is for.
• One GPU, no redundancy. A single replica means a firmware update takes a regulated workflow offline. N+1 is not optional for production.
• Skipping the license read. The Llama 3.1 community license permits commercial use but requires "Built with Llama" attribution on redistribution and imposes an acceptable use policy; organizations above 700 million monthly active users need a separate license from Meta. That threshold is irrelevant for a bank, but your legal team should still read the license on the model card before deployment.

When private deployment is the wrong choice

Honest limits, because buying hardware is hard to undo:

• Your volume is low or spiky. Below a few hundred million tokens a month, per-token API pricing beats owned hardware on cost. Run the numbers before falling in love with a server.
• You need frontier-model quality. Claude models cannot be self-hosted or fine-tuned by customers; Anthropic offers deployment options but not weights. If your workload needs that quality tier, the private path means accepting the current open-weight capability level (Llama, Qwen, Mistral, Gemma), which is strong but not identical. Our Claude for financial services page covers what the managed path looks like when that quality bar is the deciding factor.
• You have no one to operate it. A GPU node without a platform engineer is a very expensive space heater. Managed or VPC deployment transfers this burden.
• Residency is your only concern. If the requirement is data staying in a jurisdiction or a virtual private cloud rather than on your own premises, a VPC deployment or a managed private endpoint solves it without hardware.

Frequently asked questions

How many GPUs do I need to run a 70B model for 200 employees? Usually one 80 GB GPU with the model at 4-bit precision, serving 20 to 40 concurrent sessions, plus a second replica for redundancy. Two hundred employees almost never means 200 concurrent sessions; measure peak concurrency during the rental pilot phase and size to that.

Is 4-bit quantization accurate enough for bank documents? Often yes, but verify on your own eval set. AWQ, the method behind most 4-bit deployments, preserves accuracy by protecting the 1 percent of weights that matter most, and it won the MLSys 2024 best paper award. Score your real tasks at BF16 and at 4-bit; if the gap is inside your tolerance, take the four-times memory saving.

vLLM or llama.cpp for production? vLLM for anything serving more than a handful of concurrent users; its continuous batching and PagedAttention deliver several times the throughput under load. llama.cpp is the right choice for pilots, single-user or edge deployments, and hardware without server-grade GPUs.

What does an eight-GPU H100 server cost in 2026? Public OEM quotes and vendor price lists put an integrated 8x H100 node at roughly $250,000 to $320,000. Renting the same capacity runs about $23,000 a month on Lambda's published rates, so break-even sits around 12 to 18 months depending on utilization and operating costs.

Can we fine-tune Claude on our own data instead? No. Anthropic does not offer customer fine-tuning of Claude models. Private fine-tuning today means open-weight models such as Llama, Qwen, Mistral, or Gemma. Our breakdown of the real cost of fine-tuning an LLM for a bank or insurer covers what that path costs.

Where to go from here

Start with the eval set and a rented GPU, not a purchase order. Two months of rental at a few thousand dollars tells you more than any sizing spreadsheet, including this one. If you want a team to run that process with you, from workload definition through hardware selection, serving stack, monitoring, and the governance documentation your supervisor will ask for, that is exactly what our enterprise AI deployment service for banks and insurers does. Talk to an engineer about your workload numbers and we will tell you, with arithmetic, whether private deployment makes sense for you.