AI Contract Review Accuracy: What the Benchmarks Show

Law firms are being sold AI contract review from three directions at once: specialist legal vendors, the big research platforms, and the general-purpose models. The honest answer on accuracy is now measurable, because 2026 produced the first independent benchmarks that test AI review against practicing lawyers on real contracts. The short version: the best models catch about three-quarters of the errors in a final contract read-through, pull out standard terms at 80 to 84 percent accuracy, and cost less than a cup of coffee per contract. They are good enough to change the economics of review, and not good enough to be the last pair of eyes.

This post pulls together what the benchmarks actually measured, what the tools cost, and how to set up a review workflow that holds up when a partner, a client, or a court asks how the work was done.

What the benchmarks actually measured

Until recently, every claim about AI contract review came from a vendor demo. Three 2026 studies changed that, and they measure different things, so it is worth being precise.

Final proofreading: the ContractScrub benchmark

In August 2026, researchers published ContractScrub, the first benchmark for contract scrubbing, the final review pass where a lawyer hunts for misused defined terms, broken cross-references, wrong party names, and contradictions. Experienced lawyers hand-planted errors across these categories, then ten frontier models were tested on catching them.

The results were humbling. Only one model reached 0.75 macro average recall, and this on a task the researchers note looks like it should suit LLMs perfectly: long documents, consistency checking, named entity recognition. Per the Legal Funding Journal summary of the study, GPT-5.5 caught 75 percent of errors, Gemini 3.1 Pro caught 74 percent, Claude Sonnet 4.6 caught 69 percent, and Claude Opus 4.7 caught 62 percent. Turning on the model's reasoning mode added 9 to 11 points, at the cost of speed.

On a 60-page agreement, a 75 percent catch rate means the model misses some planted errors. Every time. That number should anchor every decision you make about workflow design.

Extraction: LLMs vs a small domain-trained model

A May 2026 paper, "A Few Good Clauses", tested models on pulling 26 standard fields out of contracts: dates, parties, termination clauses, liability caps. Among the frontier models, Gemini 3.1 Pro and Claude Opus 4.6 both scored 82 percent, Claude Sonnet 4.6 scored 80 percent, and GPT-5.4 scored 78 percent.

The more interesting finding: a small, domain-trained, self-hosted model (Olava Extract, built by the study authors' employer, so apply the usual discount) hit 84 percent with a macro F1 of 0.812 while cutting inference cost by 78 to 97 percent, and it produced the fewest hallucinated extractions. For high-volume extraction work, a narrow model you host yourself can beat the flagship you rent. The same build-versus-rent arithmetic applies outside legal, as we covered in AI Agents for Document Processing Automation.

The vendor benchmark worth reading anyway

Ivo, a contract review vendor, commissioned a blind comparison of its own product, Claude for Word, and a practicing special counsel at an Am Law 25 firm. Three attorney judges scored anonymized redlines of 19 real contracts (NDAs, MSAs, DPAs) without knowing which reviewer produced which.

The scores: the human lawyer 4.56, Ivo 4.52, Claude for Word 3.50. The human took about 10 hours for the 19 contracts; Ivo averaged 2 minutes 45 seconds per contract and Claude just under 5 minutes. Ivo paid for the study, so treat the headline accordingly, but the methodology (blind scoring, real contracts, a real lawyer baseline) is more serious than most vendor claims, and the speed-versus-quality trade-off it exposes is real.

The numbers that matter

Condensed from the studies above, here is what a buyer should hold onto:

• Final proofreading recall: 62 to 75 percent of planted errors caught, depending on model, with reasoning mode adding 9 to 11 points.
• Extraction accuracy: 78 to 84 percent across 26 standard fields for the best models.
• Cost per contract: $1.38 and about nine minutes for GPT-5.5 with reasoning; $0.19 and about 90 seconds for Gemini 3.1 Pro.
• Human pace: roughly 32 minutes per contract in the Ivo test (10 hours across 19 contracts), at lawyer rates.
• Precision matters too: the Legal Funding Journal notes Claude models were the most careful about not flagging problems that were not there, which cuts the review burden of false positives.
• Never from memory: asked to recall exact case citations without source documents, the best model in one 2026 study scored under 7 out of 100, and even the strongest models cited something false 15 to 17 percent of the time. Grounded in the actual documents, fabrication rates dropped to 4 to 15 percent, and to under 0.2 percent with a well-built retrieval system.

That last point generalizes beyond research memos: a model working from the contract in front of it is a different, far safer tool than a model working from what it remembers.

General model or specialist tool

The studies support a three-way split.

A general model with a good setup (Claude in Word with tracked changes, or GPT-5.5 with reasoning on) handles first-pass review, extraction, and summarization at a fraction of a cent on the dollar. It loses to specialists on surgical redlining and judgment calls, per the Ivo study, and it needs you to supply playbooks and checklists.

A specialist contract tool (Ivo, Spellbook, Luminance) earns its subscription when redlining quality against your playbook is the bottleneck and volume is steady. You are paying for the playbook layer, the Word integration, and tuning, not for a fundamentally different brain.

A self-hosted extraction model starts to make sense at high volume or when documents cannot leave your environment. The "A Few Good Clauses" results show a narrow model can beat frontier models on structured extraction at a fraction of the inference cost, and private deployment keeps privileged material inside your walls. This is the same build-versus-rent decision insurers face on claims triage; we covered the trade-offs in LLM Claims Triage for Insurers: Accuracy, Human Review, Cost.

One more option sits between these: Anthropic publishes Claude for Legal, an open-source suite of 12 practice-area plugins with skills for contract work and connectors into systems like iManage, DocuSign, Ironclad, and CourtListener. The plugins themselves carry no license fee; you pay for the Claude plan. That narrows the gap between a general model and a specialist tool for firms already standardized on Claude, and it is the stack our Claude for Legal implementation work is built around.

What the tools actually cost

Most legal AI vendors hide their pricing. Based on published figures and third-party estimates (flagged where they are estimates):

• Claude (general model): self-serve plans with published pricing in the tens of dollars per seat per month; the Claude for Legal plugin suite is free and open source.
• Clio Duo: published, starting at $49 per user per month.
• CoCounsel (Thomson Reuters): third-party estimates around $293 to $492 per user per month, with a 7-day trial.
• Spellbook (contract drafting and review in Word): quote only, priced by team size.
• Ironclad (contract lifecycle management): reported median around $39,995 per year with a $15,000 minimum annual contract.
• Harvey: no public pricing; third-party estimates put it at roughly $1,200 to $2,800 or more per seat per month with a seat minimum around 20, which implies a floor near $288,000 a year. Treat these as market estimates, not quotes.

The spread is the point. A five-lawyer commercial team can be running real AI-assisted review for under $300 a month in seat costs. The enterprise platforms are priced as labor substitutes for large firms, and several simply do not sell to small ones.

Worked example: a commercial team reviewing 40 contracts a month

Take a firm with a commercial team that reviews 40 contracts a month: NDAs, MSAs, DPAs, vendor agreements. Blended lawyer cost of $300 an hour.

Human-only review. At the pace measured in the Ivo study (about 32 minutes per contract), that is roughly 21 lawyer-hours a month, or about $6,300 in lawyer time.

AI first pass, lawyer verifies. Run each contract through a frontier model with reasoning on. At the ContractScrub-measured cost of $1.38 per contract, that is $55 a month in model spend. The lawyer then verifies the AI markup instead of reading cold: 15 focused minutes per contract instead of 32, which is 10 hours, or $3,000. Total: roughly $3,055 a month.

The catch is in the word "focused." The model catches about three errors in four. The 15 minutes of verification is not a skim; it is the lawyer checking the categories the model handles worst (defined terms, cross-references) and every change the model proposed. If verification degrades to a two-minute glance, you have not saved $3,200 a month. You have bought a 25 percent miss rate and taken personal responsibility for it.

The same arithmetic scales to due diligence, where contract volume spikes into the hundreds. We walked through that workflow in AI for Private Equity Due Diligence: Workflow, Costs, and Limits.

A review workflow that survives scrutiny

Here is the procedure that lines up with both the benchmark evidence and the ethics rules:

1. Feed the model the documents, always. Never ask it to recall contract language, market terms, or law from memory. Attach the contract, your playbook, and any fallback positions. Grounding the task in source documents is the single biggest error reducer the research found.
2. Turn reasoning mode on for review tasks. It costs minutes and adds 9 to 11 points of error-catching. For a contract, that trade is never close.
3. Require the model to quote its basis. Every flag should cite the clause it relies on. Unverifiable flags get deleted, not debated.
4. Use a checklist for the model's blind spots. Defined terms, cross-references, party names, dates: the categories ContractScrub tested are exactly where models slip, so the human verification pass should start there.
5. Keep a named human accountable for output. The model is the first pass and the tireless second reader. It is never the final reviewer on anything that goes to a client, a counterparty, or a court.
6. Route approvals explicitly. If juniors run the first pass, route AI-assisted markups to a reviewer with sign-off authority. We described how to wire that kind of human checkpoint into an automated flow in n8n Approval Workflows: Human Review for AI Agents in Slack.
7. Test before you trust, and retest on model upgrades. Take 20 contracts your firm has already reviewed, run them through the tool, and count what it catches and what it invents. Rankings shift with every model release; your own documents are the only benchmark that matters.
8. Record what was used. Log which tool, which model version, and who verified, per matter. This takes minutes and answers the questions a court or client may ask later.

If the setup work (playbooks, plugins, connectors into your document management system, verification logging) is more than your team wants to own, that is the piece our Claude implementation service handles: configuring the stack, building the checklists into skills, and training the team on the workflow above.

The ethics duties you cannot delegate

ABA Formal Opinion 512 (July 2024) sets the frame, and state bars have followed it. For contract review, four duties bite hardest:

• Competence (Rule 1.1): you must understand the capabilities and limits of the specific tool you use. Reading the benchmarks in this post is part of that duty, not optional homework.
• Confidentiality (Rule 1.6): you must evaluate the risk that client information is disclosed to or through the tool. That means reading the vendor's data terms, and it is the strongest argument for self-hosted models when documents are genuinely sensitive.
• Candor and supervision (Rules 3.1, 3.3, 5.1, 5.3): the lawyer is fully responsible for the output regardless of how it was produced. "The AI missed it" is not a defense; it is the admission.
• Reasonable fees (Rule 1.5): you can bill for the time you actually spend prompting and reviewing, but you cannot bill for hours not worked, and charging the old flat fee for work AI now does in a fraction of the time may itself be unreasonable.

The sanctions cases show what happens when verification fails. In Deutsche Bank National Trust Co. v. LeTennier, covered by the New York State Bar Association, an appellant's opening brief cited six nonexistent cases, and later filings added more fabricated authority. Courts now treat hallucinated citations as sanctionable conduct, and researchers counted more than 1,000 court filings containing fabricated citations, a number growing every year. Most of those involve research memos and briefs rather than contracts, but the underlying failure is the same: output that was never checked by the person who signed it.

When AI contract review is the wrong choice

Be suspicious of anyone who sells this as universal. It is the wrong choice when:

• Volume is low. Under about ten contracts a month, the setup and verification discipline cost more than they return. Just review them.
• The agreement is genuinely novel. A first-of-its-kind joint venture or a structured finance deal has no pattern for the model to match against. Benchmarks tested standard commercial agreements, and that is where the tools earn their keep.
• You cannot verify. If nobody on the team has the expertise to catch what the model misses, automating first-pass review just moves errors faster.
• Data terms do not work. If the vendor's confidentiality terms fail your Rule 1.6 analysis and you are not prepared to self-host, the answer is no until one of those changes.

Common mistakes

• Buying on vendor benchmarks alone. Even the well-run ones (Ivo's blind study is better than most) are commissioned by someone selling something. Weight independent studies more heavily, and run your own 20-contract test.
• Asking from memory. The single largest error source in the 2026 research. Attach the documents, every time.
• Leaving reasoning mode off to save time. You save eight minutes and lose 9 to 11 points of accuracy. On a contract, that is a bad trade.
• Letting verification decay into a skim. Put the 15 minutes on the matter plan. If reviewers routinely spend two minutes, the workflow is broken, not the model.
• Rolling out firm-wide on day one. Pilot on one practice group and one contract type, measure catch rates against contracts already reviewed, then expand.
• Ignoring the fee question. If AI cuts a four-hour review to one hour of verification, your billing model has to change before a client changes it for you.

Frequently asked questions

How accurate is AI contract review? On independent 2026 benchmarks, the best general models caught 62 to 75 percent of errors planted by lawyers in final contract review, and extracted standard terms with 78 to 84 percent accuracy. In one vendor-commissioned blind study, a specialist tool scored nearly level with an Am Law lawyer on standard commercial agreements. Accurate enough to do the first pass; not accurate enough to skip verification.

Is Claude good enough for contract review? Claude models sit mid-pack on raw error-catching (Claude Sonnet 4.6 caught 69 percent of planted errors in ContractScrub) but were the most careful about false positives, which lowers the cost of human verification. With the free Claude for Legal plugins and tracked changes in Word, it works well as a first-pass reviewer inside a supervised workflow.

How much does AI contract review cost? From tens of dollars per seat per month for a general model with legal plugins, to an estimated $293 to $492 per user for CoCounsel, to roughly $1,200 to $2,800 per seat per month with a 20-seat minimum for Harvey (third-party estimates; Harvey publishes no pricing). Model API costs for the review itself are trivial: $0.19 to $1.38 per contract in the benchmarks.

Can I bill clients for AI-assisted contract review? Yes, for time actually worked. ABA Formal Opinion 512 allows billing for the time spent prompting the tool and reviewing its output, but not for hours you did not expend, and it warns that an unchanged flat fee may be unreasonable if AI made the work much faster.

Do AI contract review tools keep client data confidential? That depends on the vendor's data terms, and Rule 1.6 makes evaluating them your job, not the vendor's. Where the terms do not work, a self-hosted open-weight model keeps documents inside your environment; independent testing shows a domain-trained small model can beat frontier models on contract extraction at far lower cost.

Will AI replace contract reviewers? The 2026 evidence says no for the foreseeable future: the best model misses a quarter of planted errors on final review. It replaces the cold read, not the judgment. The firms that benefit are the ones that redeploy the saved hours into the verification and negotiation work the models cannot do.

Where to go from here

The benchmarks point to a clear pattern: ground the model in your documents, turn reasoning on, verify with a checklist aimed at the model's known blind spots, and keep a named lawyer accountable for everything that leaves the building. If you want that workflow running on Claude with the legal plugins, playbooks, and connectors configured for your practice, our Claude for Legal implementation is built exactly for that. Start with the 20-contract test above; it will tell you more than any vendor demo.