How to Choose a Speech-to-Text API for a Voice Agent
A numbers-first comparison of the leading speech-to-text APIs for voice agents: benchmark accuracy, accented speech, latency, per-call cost, self-hosting Whisper, and a one-week evaluation procedure.

A voice agent lives and dies on its transcript. If the speech-to-text layer mishears the caller, the language model answers the wrong question, and no amount of prompt engineering fixes that. This guide compares the leading speech-to-text APIs on the three things that decide whether a voice agent works: accuracy on real and accented speech, latency, and cost per call. It ends with a one-week procedure for testing the candidates on your own audio, because the benchmark numbers vendors publish disagree with each other in ways that matter.
What a voice agent actually needs from speech-to-text
Transcribing a podcast and transcribing a live phone call are different jobs. Before comparing models, be clear about what a production voice agent needs from the API.
- Streaming partials. The agent cannot wait for the caller to finish a paragraph. It needs interim transcripts every few hundred milliseconds so the LLM can start reasoning early.
- Endpointing. The API must detect when the caller has finished talking. Get this wrong and the agent either interrupts people or leaves dead air.
- Accuracy on degraded audio. Phone calls arrive at 8 kHz with background noise, crosstalk, and compression artifacts. Real-world accuracy on phone conversations runs 80 to 88 percent, against 95 to 98 percent for clean studio audio, per AssemblyAI's accuracy analysis.
- Accents and code-switching. If your callers mix languages or speak with accents the training data underrepresents, headline benchmark numbers will not predict your results.
- Domain vocabulary. Product names, branch names, policy numbers, medical terms. Some APIs support keyterm prompting or custom vocabulary; others do not.
- Diarization. For analytics on recorded calls you need speaker separation. For a live agent you mostly do not.
Keep this list in mind. Several of the numbers below look decisive until you check whether they measure the thing you need.
How accurate are the leading APIs
The benchmark numbers vendors publish
AssemblyAI runs a public benchmark with a documented methodology: fixed audio sets, default settings, error rates recomputed each cycle. Its pre-recorded English results, from runs in August and September 2026:
- AssemblyAI Universal-3.5 Pro: 4.35 percent word error rate (WER), at $0.21 per hour
- OpenAI GPT-4o Transcribe: 5.34 percent WER, at $0.36 per hour
- ElevenLabs Scribe V2: 5.87 percent WER, at $0.40 per hour
- Deepgram Nova-3: 6.66 percent WER, at $0.26 per hour
- Azure Batch: 7.02 percent WER, at $0.36 per hour
This is a vendor-run benchmark, so treat the ordering with appropriate caution, but the audio sets and metric definitions (WER, named-entity error rate, medical dictation MER, diarization cpWER) are published and the price-per-hour figures are useful regardless of who ranked first.
For streaming, the numbers change. The independent Artificial Analysis streaming speech-to-text leaderboard measured:
- AssemblyAI Universal-3.5 Realtime Pro (max accuracy): 4.03 percent WER at $7.50 per 1,000 minutes ($0.45 per hour)
- OpenAI GPT Realtime: 4.89 percent WER at $17 per 1,000 minutes ($1.02 per hour)
- Deepgram Nova-3 Realtime: 6.59 percent WER at $4.80 per 1,000 minutes ($0.29 per hour)
- Deepgram Flux: 7.39 percent WER at $6.50 per 1,000 minutes
Why the numbers disagree
Deepgram's own API comparison claims Nova-3 scores 5.26 percent WER on general English. AssemblyAI measured the same model at 6.66 percent. Both can be honest: WER is a property of a model plus a test set, and neither vendor uses the other's audio. Accents, sample rates, audio quality, and even how punctuation is normalized all move the number by points.
Two consequences follow. First, never compare two models using numbers from two different benchmarks. Second, the only benchmark that predicts your agent's behavior is one run on your callers' audio. More on that in the evaluation procedure below.
One more honest limit from the AssemblyAI data: diarization is bad everywhere. The best cpWER (concatenated minimum-permutation word error rate, the standard speaker-attributed metric) in that table is 30.17 percent, and Deepgram scores 37.93 percent. If your use case is analytics over recorded two-party calls, budget for speaker attribution errors or use telephony that records each leg on its own channel.
The accent gap nobody benchmarks
Every mainstream benchmark runs on American and British English. That is a problem if your callers are in Nairobi, Lagos, or Mumbai, and it is the single largest gap in the public data.
The clearest evidence is the AfriSpeech-200 benchmark, a peer-reviewed dataset of 200.7 hours of African-accented English: 67,577 clips from 2,463 speakers across 120 accents in 13 countries. The finding that should drive your vendor selection: leading models perform three to ten times worse on African-accented speech than on the standard LibriSpeech test set. A model at 5 percent WER on clean American audio can be at 20 percent or worse on an accented caller.
Vendor data tells the same story. In AssemblyAI's accented-English benchmark, ElevenLabs Scribe v2 scores 7.0 percent WER and Universal-3.5 Pro 7.1 percent, while Deepgram Nova-3 scores 14.4 percent. Broken out by accent group, Nova-3 reaches 20.3 percent on French-accented and 19.8 percent on Spanish-accented English, against 9.3 percent on British and 8.2 percent on Indian accents. AssemblyAI's own real-world guidance puts heavily accented speech at 75 to 90 percent accuracy overall.
Put that in concrete terms. In a 75-word caller utterance, a 4.35 percent WER means about 3 wrong words, and the agent will usually recover. A 14.4 percent WER means about 11 wrong words, and the LLM downstream is now reasoning over a corrupted input. This is how agents end up confirming the wrong account number or booking the wrong date.
The good news from AfriSpeech-200: fine-tuning closes much of the gap. Fine-tuning Whisper-medium on accented data cut WER by 32.6 to 34.9 percent relative, and fine-tuned open models beat the commercial APIs tested by 36.5 percent on average across the strongest accents. If your call volume is high and your callers share a small set of accents, a fine-tuned open model is a serious option, not a science project. We cover the Kenya-specific version of this in how to test a voice AI agent on Kenyan speech before launch and choosing an ASR model for Swahili and Kenyan English.
Latency and turn-taking
For a phone agent, raw accuracy is only half the spec. Sub-300-millisecond response latency is now table stakes for conversational speech APIs, as Deepgram's comparison notes, and the harder problem is turn-taking: deciding the caller has finished. A model that transcribes perfectly but takes an extra second to finalize forces awkward pauses into every turn.
Two things to look at when testing:
- Time to final transcript after the caller stops speaking, not just time to first partial. The LLM waits for the final.
- Endpointing design. Deepgram's Flux bakes end-of-turn detection into the model itself; other stacks estimate it from silence timeouts plus transcript semantics. Silence-only endpointing cuts off slow, hesitant speakers.
Latency is a system property, not a model property: telephony hops, network routing, and the LLM's own response time dominate the budget. Our post on where the milliseconds go on voice AI phone calls breaks down the full path.
What speech-to-text actually costs
STT is the cheapest layer of a voice agent, which means accuracy should drive the choice more than price. Here is a worked example at a realistic volume: 8,000 calls per month, four minutes each. That is 32,000 minutes, or 533 hours of audio.
Raw streaming STT, per month:
- Deepgram Nova-3 Realtime ($0.29/hour): about $154
- AssemblyAI Universal-3.5 Realtime Pro ($0.45/hour): about $240
- OpenAI GPT Realtime ($1.02/hour): about $544
Pre-recorded transcription, for recorded-call analytics at the same volume:
- AssemblyAI Universal-3.5 Pro ($0.21/hour): about $112
- Deepgram Nova-3 ($0.26/hour): about $139
- OpenAI GPT-4o Transcribe ($0.36/hour): about $192
- ElevenLabs Scribe V2 ($0.40/hour): about $213
The gap between the cheapest and most expensive serious option here is roughly $400 a month at this volume. A single percent of failed calls costs more than that in almost any business.
Compare this with bundled voice-agent APIs, which price the whole runtime (STT, LLM orchestration, TTS) rather than transcription alone. Deepgram's Voice Agent API runs $0.075 per minute on pay-as-you-go for the standard tier, or $0.050 per minute if you bring your own LLM and TTS: $2,400 and $1,600 per month respectively at our example volume. AssemblyAI's Voice Agent API is $4.50 per hour, also about $2,400 per month, including access to Claude, GPT, and Gemini through its LLM gateway. These bundles are not comparable to raw STT prices; they replace several vendors and a good deal of integration work. Both vendors offer $200 in free credits, enough to run a real evaluation.
Two pricing details worth checking before you sign anything: Deepgram bills its agent API on websocket connection time, so idle silence costs money, and compliance add-ons (PII redaction, content moderation) are metered separately at AssemblyAI, from $0.01 to $0.15 per hour. Prices in this section come from the vendor pages and benchmark tables cited above and change often; verify against the live pricing pages before budgeting.
When self-hosting Whisper makes sense
Whisper is open-source under the MIT license, with the weights on GitHub and 111,000 stars of ecosystem behind it. Self-hosting is the right call in a clear set of situations:
- Data residency. Bank, insurer, hospital, or government audio that cannot leave your VPC or your jurisdiction. This is the most common legitimate reason, and it is a deployment problem we handle through our enterprise AI work.
- Fine-tuning on your domain. As AfriSpeech-200 showed, fine-tuned Whisper beats commercial APIs on accented speech by a wide margin. If your callers share a few accents and you can collect training audio, this is the highest-accuracy path.
- High, steady volume. Past a few hundred thousand minutes a month, a dedicated GPU can undercut per-minute API pricing, if you already run GPU infrastructure.
Self-hosting is the wrong choice when your volume is small (at 533 hours a month, the most expensive API costs $544; a suitable GPU server costs more), when you need streaming diarization and turnkey PII redaction, or when nobody on your team wants to own a GPU service. Whisper is also fundamentally a batch model: real-time phone use requires community streaming wrappers, which add engineering surface area. The honest summary: self-host for residency or fine-tuning, not to save money at low volume.
How to run your own evaluation in a week
Given that vendor benchmarks disagree and none test your callers, the only reliable comparison is one you run yourself. This procedure fits in five working days and costs under $50 in API credits.
- Collect 50 to 100 real calls. Pull them from your telephony provider. They must include your worst audio: mobile networks, speakerphone, background noise, and your actual accent mix. Get consent handling right before recording anything.
- Transcribe them by hand. At minimum, transcribe the first 60 seconds of each call. This is your ground truth. Do not skip it; every other step depends on it.
- Run each candidate API over the same files. Use default settings first, then each vendor's telephony or accuracy mode. Three candidates is enough: typically Deepgram Nova-3, AssemblyAI Universal, and one wildcard (Scribe, GPT-4o Transcribe, or a fine-tuned Whisper).
- Compute WER per call. The standard tool is the
jiwerPython package:
pip install jiwer
from jiwer import wer
score = wer(reference_transcript, api_transcript)
- Score what actually breaks agents. Overall WER is a screening metric. Separately tally errors on the entities your agent acts on: names, account numbers, amounts, dates. A model can win on WER and lose on the numbers that matter.
- Measure streaming latency. For your top two, run a live streaming test: time from end of caller speech to final transcript, across at least 30 utterances, and record the 95th percentile, not the average.
- Decide with a checklist. Pick the model that clears all of these: WER on your audio within two points of the best candidate, entity error rate under your tolerance, 95th-percentile finalization latency under one second, pricing that works at ten times your current volume, and a data-processing agreement your compliance team accepts.
If you would rather hand this to a team that builds voice agents for a living, this evaluation is part of how we scope custom builds.
Common mistakes when choosing a speech-to-text API
- Comparing numbers from different benchmarks. A 5.26 percent claim and a 6.66 percent measurement can both be true on different audio. Only same-audio comparisons count.
- Testing on clean audio. Vendor demo clips are not your phone line. Eight-kilohertz telephony audio with compression artifacts is a different, harder input.
- Ignoring accents until after launch. The three-to-ten-times WER penalty on underrepresented accents is the most common way voice agents fail in production, and it never shows up in a demo tuned to American English.
- Optimizing price before accuracy. At realistic volumes the entire STT line item is a few hundred dollars a month. One percent of mishandled calls costs more.
- Trusting silence-based endpointing. Hesitant, elderly, or non-native speakers pause mid-sentence. Test endpointing on them specifically.
- Forgetting diarization is unsolved. Speaker-attributed error rates of 30 percent or more are normal. Architect around it for recorded-call analytics.
Frequently asked questions
What is a good word error rate for a voice agent?
Under 8 percent on your own call audio is workable, because the LLM can tolerate some noise. Under 5 percent is good. But test entity accuracy separately: an agent can survive a wrong filler word and fail on a wrong digit.
Is Deepgram or AssemblyAI more accurate?
On clean American English they are close, with AssemblyAI ahead in both its own benchmark and the independent Artificial Analysis streaming table. On accented English the gap widens considerably in AssemblyAI's accented benchmark. On your audio, you have to measure it yourself.
Does speech-to-text work on African accents?
Out of the box, poorly: the AfriSpeech-200 study found models run three to ten times worse on African-accented speech than on standard benchmarks. Fine-tuned open models cut that gap by about a third and beat commercial APIs on the tested accents, so for African call volumes, plan for either a fine-tuned model or a vendor that scores well on your own test set.
Can I use Whisper for a real-time phone agent?
Yes, with caveats. Whisper is a batch model, so you need a streaming wrapper, your own GPU, and your own endpointing logic. It makes sense for data residency or fine-tuning, not for saving money at low volume.
How much does speech-to-text cost per call?
At streaming API prices of $0.29 to $1.02 per hour, a four-minute call costs between roughly two and seven cents to transcribe. Bundled voice-agent runtimes that include the LLM and text-to-speech run 30 to 45 cents for the same call.
What is the difference between WER and diarization error?
WER measures how many words were transcribed wrong. Diarization error (cpWER) additionally penalizes words attributed to the wrong speaker. The best current systems score around 30 percent cpWER, which is why speaker-attributed analytics still needs human review.
Where to go from here
The short version: ignore cross-vendor benchmark claims, shortlist Deepgram Nova-3, AssemblyAI Universal, and one wildcard, and run the one-week evaluation on your own call recordings before you commit. If the callers you care about speak with accents the benchmarks ignore, weight that test set heavily; it is where every public number is weakest. And if you want a voice agent designed, tested, and run for you rather than assembled in-house, our AI Employees service delivers an always-on agent built on proven open-source infrastructure, with the speech stack evaluated and tuned for your callers from day one.
About AI Agents Plus Editorial
AI automation expert and thought leader in business transformation through artificial intelligence.



