Voice AI Latency Budget: Getting an Agent Under One Second

A voice agent that answers one and a half seconds late feels broken, no matter how good the answer is. The fix is not a faster model; it is a per-stage latency budget, measured end of caller speech to first audio byte out, with a target under 800 milliseconds and a hard cap around 1,200. This post gives you that budget, the tuning levers for each stage, and the measurement procedure to know whether you are actually hitting it.

The numbers below come from vendor documentation, production engineering write-ups, and one open-source latency budget guide. Where figures come from a single vendor's production system, that is stated.

What callers actually perceive

Human conversation runs on a surprisingly tight clock. A study of ten languages published in PNAS (Stivers et al., 2009) found that speakers everywhere avoid overlapping talk and minimize silence between turns, with the average gap between turns clustering around 200 to 250 milliseconds. Your callers are calibrated to that rhythm from a lifetime of conversations. An agent that routinely answers in 1.5 seconds or more reads as unresponsive, and turn-taking research attributed to Google puts the "unresponsive" threshold right around there.

So the perceptual targets are:

• Under 800 ms from end of caller speech to first audio byte: feels natural. This is the target used in the open-source voice-agent-kit latency budget guide, with 1,200 ms as a hard cap.
• Around 1,500 ms and up: callers start wondering if the line dropped, say "hello?", or talk over the agent's late reply.

One caution before the engineering: latency is a floor, not a finish line. Full-duplex benchmarks published in 2026 show leading real-time voice models in the same latency class with interruption rates from roughly 13.5% to 47.9%. One model cuts the caller off almost half the time; another does it about one call in seven. Same speed, wildly different manners. Getting under a second matters. Getting the turn-taking right matters more, and we cover that below.

The anatomy of one phone turn

On a phone call, the budget is tighter than in a web demo. Cisco's Webex AI Agent engineering team reports that the PSTN call path itself introduces roughly 500 ms of latency, before any AI processing happens. Audio typically reaches your stack through a bidirectional WebSocket such as Twilio Media Streams, which streams raw call audio to your server and plays your audio back into the call.

A standard cascaded pipeline then runs these stages on every turn:

• Voice activity detection and endpointing: deciding the caller has finished speaking.
• Speech-to-text (STT): producing the final transcript.
• LLM reasoning: generating the reply, often with retrieval or tool calls.
• Text-to-speech (TTS): synthesizing the first chunk of audio.
• Network and telephony return path: getting that audio back to the caller's ear.

We break down where each millisecond goes across the whole call path in Voice AI Latency on Phone Calls: Where the Milliseconds Go. Here the focus is the budget: what each stage is allowed to spend.

Cisco's published production breakdown shows what a heavily optimized system looks like. Their VAD end-of-speech fires at about 500 ms of silence, turn detection adds under 75 ms at p99, and the first segment of the answer has already been generated and cached while the caller was still speaking, so TTS on that segment takes 10 to 20 ms. The result is roughly 1.3 seconds end to end over real telephony, which is respectable on PSTN. The key insight: their best trick is not faster components, it is doing the work earlier, in parallel with the caller's own speech.

A per-stage latency budget that works

For a self-hosted or vendor-assembled pipeline, here is a concrete 800 ms compute budget, using P50 figures from the voice-agent-kit provider measurements:

• Endpointing and VAD: 300 ms. Silence threshold before treating the turn as done. This is a deliberate wait, not processing time, and it is the biggest single dial you control.
• STT final transcript: 150 ms. Deepgram Nova streaming runs around 100 ms at P50 (250 at P99); AssemblyAI is similar; AWS Transcribe is closer to 150 ms P50.
• LLM time to first token: 250 ms. What matters is first token, not full completion. With streaming enabled, the first sentence of the reply can go to TTS while the rest is still generating.
• TTS first audio byte: 150 ms. Cartesia Sonic reports around 80 ms P50, Deepgram Aura around 100 ms; ElevenLabs Turbo and Google TTS sit near 150 ms; AWS Polly Neural and OpenAI TTS are closer to 200 ms.
• Total: 850 ms before network, with the PSTN path adding its own fixed cost on top.

That budget holds at the median. At P99 every stage gets worse: STT runs 250 ms, tool calls or retrieval can push the LLM stage to 600 ms, TTS hits 250 ms. This is why the guide recommends alerting on P95 above 800 ms, P99 above 1,200 ms, and more than 5% of turns exceeding budget, rather than on averages. Averages hide the tail, and the tail is what callers remember.

Tuning levers, stage by stage

Transport and telephony

• Colocate everything in one region. Cross-region calls add 50 to 100 ms per hop. Put your media server, STT, LLM proxy, and TTS in the same region.
• Keep connections warm. AWS Polly has a roughly 200 ms cold start on the first request, Google Cloud TTS around 150 ms. Persistent WebSockets (Deepgram has effectively no cold start) and connection pools remove this.
• Use bidirectional streaming telephony. Twilio's <Connect><Stream> or an equivalent SIP/WebRTC path, not record-then-process.

Speech-to-text

• Enable interim results so downstream logic can start on partial transcripts instead of waiting for the final one.
• Tune endpointing explicitly. Deepgram's endpointing parameter is the milliseconds of silence before a transcript is finalized; it defaults to a hair-trigger 10 ms and is commonly set between 300 and 500 ms. Pair it with utterance_end_ms and VAD events rather than relying on a single signal.
• Pick a streaming model, not batch. Batch HTTP transcription (plain Whisper over HTTP) costs 200 to 800 ms before you have said a word.

LLM

• Stream and ship the first sentence early. Time to first token is the only number that matters for latency; the rest of the reply generates while TTS is already speaking.
• Cache the prompt. System prompts, tool definitions, and knowledge snippets that repeat across turns should hit prompt caching, which Cisco lists among its biggest wins alongside reserved model capacity.
• Hedge slow requests. Cisco runs parallel "safety-net" requests to guard against a slow LLM response, plus an LLM proxy with regional failover. Expensive, but it kills the P99 tail.
• Pre-generate where you can. Cisco's most effective optimization is generating the opening of the answer while the caller is still talking, then continuing from it once end of turn is confirmed. The early segment is never discarded; the final answer simply continues. This is what lets them run large, accurate models while still answering fast.

Text-to-speech

• Stream synthesis and start playback on the first chunk, not the full utterance.
• Skip container overhead. Deepgram's own tuning note recommends container: 'none' to avoid WAV header delay; match encoding to your telephony (pcm_mulaw for Twilio).
• Pre-synthesize fixed lines. Greetings, hold messages, and disclosures are known in advance. Synthesize them at deploy time and the TTS stage costs 10 to 20 ms on those turns, which is exactly what Cisco's cache-hit numbers show.

Endpointing: the fastest agent is not the right one

The single most under-tested decision in a voice stack is endpointing: judging whether the caller is done speaking or merely pausing. A caller who says "yeah, one sec, let me find it" and goes quiet for four seconds while looking for an account number will be interrupted by an agent tuned to fear dead air. Now the agent and the caller are talking over each other, and a call a human receptionist handles without thinking has become a fight over whose turn it is.

The fix is context-dependent endpointing. The pause after "my number is..." should be long, because digit strings come with pauses. The pause after "yes, that works" should be short, because the caller is done. Three practical ways to get there:

• A turn-detection model. LiveKit's turn detector predicts end of turn from both the meaning of the speech and its acoustic properties, on top of VAD, with configurable minimum and maximum endpointing delays and dynamic endpointing that adapts the delay to context.
• STT-side signals. Deepgram's endpointing, utterance-end, and VAD events give you several signals to combine rather than one global silence timer.
• Semantic checks. Cisco's business-logic layer uses entity-aware rules, for example treating digit sequences differently from confirmations.

Whatever you choose, measure the interruption rate as a first-class metric: how often the agent starts talking while the caller is still talking. The newer academic benchmark Full-Duplex-Bench-v3 scores agents specifically under disfluency, the ums, mid-sentence restarts, and pauses to read a number, because that is where polished demo numbers fall apart. Your evaluation set should include those turns deliberately. A test set of clean, complete sentences measures the one condition your agent will never actually face.

Pipeline or speech-to-speech

There is a real architectural choice here, and it is a trade, not a free win.

Speech-to-speech models such as OpenAI's Realtime API or Gemini Live take audio in and produce audio out in one model, skipping the separate STT and TTS stages. Measured P50s from the voice-agent-kit guide are around 300 ms for OpenAI Realtime and 250 ms for Gemini Live, competitive with a well-tuned pipeline, with more natural prosody and native interruption handling over WebRTC, WebSocket, or SIP.

The cost is observability and control:

• No transcript seam. In a cascaded pipeline, every turn has a text transcript and a stage boundary you can inspect. When the agent interrupts a customer, you can open the call and see why. A speech-to-speech black box takes that away.
• Harder grounding and audit. RAG, entity checks, and compliance logging all live at the text layer. Banks, insurers, and law firms that need an auditable record of what was said and why the agent answered as it did should think carefully before giving that up.
• Less component freedom. A pipeline lets you swap STT, LLM, or TTS independently as prices and quality move. Speech-to-speech is one vendor, one model.

For most business deployments, a well-tuned pipeline with a strong text model at the center is the better default: it hits the same perceptual latency class and keeps the transcript you will need for debugging, evaluation, and governance. This is also the architecture we reach for in our Claude implementation work, where Claude handles reasoning and tool use over MCP while specialized services handle the audio. Speech-to-speech earns its place for open-ended, prosody-sensitive conversation where no audit trail is required.

How to measure your own agent

Vendor latency claims are measured on their audio, their network, their demo script. Run this procedure on your own calls before believing any of them:

1. Instrument every stage with timestamps. End of caller speech, final transcript, LLM first token, TTS first byte, audio out to telephony. Five timestamps per turn.
2. Compute per-stage and end-to-end histograms. Report P50, P95, and P99 for end-of-speech to first audio byte. Never average.
3. Set alerts at the tail. P95 above 800 ms, P99 above 1,200 ms, budget-exceeded rate above 5%, per the voice-agent-kit thresholds.
4. Record the interruption rate. Count turns where agent audio overlaps caller speech. Track it next to latency, not as a footnote.
5. Test with disfluent audio. Build a set of recorded turns with four-second pauses, restarts, and digit strings read slowly. If your agent survives that set, it will survive real callers.
6. Re-measure after every change. A new STT model, a new region, a bigger system prompt: each one moves the tail.

Common mistakes

• Benchmarking on demo-clean audio. Clean sentences, no pauses, no background noise. The benchmark passes and the first real caller reading an account number breaks the agent.
• One global silence timeout. Too short interrupts callers mid-thought; too long leaves dead air. Endpointing must vary with context.
• Optimizing the average. A 600 ms average with a 2.5-second P99 means one turn in a hundred feels dead, and callers hit several of those per call.
• Opening fresh connections per call. Cold starts of 150 to 200 ms on TTS and re-handshakes on every call are pure waste. Keep WebSockets and pools warm.
• Chasing the 200 ms headline. Below the perceptual floor, extra speed buys nothing the caller can hear, and a hair-trigger agent starts treating every human pause as its turn.
• Ignoring the phone network. PSTN adds its own latency budget before your code runs. A web demo that feels instant can feel sluggish on a real call.

When latency is the wrong priority

Latency work is premature in two situations. First, if the agent's answers are wrong, ungrounded, or unable to complete the task, fix that first; a fast wrong answer is still a wrong answer, and the cost side of the build is covered in our voice AI implementation cost breakdown. Second, once you are reliably under about 800 ms at P95, further optimization has diminishing returns that callers cannot perceive. Spend that effort on turn-taking quality, interruption rate, and answer accuracy instead. The voice AI customer service implementation guide covers how to sequence the whole build so latency work lands in the right order.

Frequently asked questions

What is a good latency for a voice AI agent? Under 800 ms from the end of the caller's speech to the first audio byte of the reply is the widely used target, with 1,200 ms as a hard cap. Human turn gaps cluster around 200 to 250 ms, so anything under about a second feels conversational; past 1.5 seconds callers start treating the agent as unresponsive.

Why does my voice agent keep interrupting callers? Almost always an endpointing problem, not a speed problem. The agent treats any silence longer than its fixed threshold as end of turn, so a caller who pauses to find a number gets cut off. Fix it with a turn-detection model or context-dependent endpointing, and track interruption rate as a metric.

Is speech-to-speech faster than a pipeline? Slightly, at the median: measured P50s run around 250 to 300 ms for the major speech-to-speech models versus a similar figure for a tuned pipeline. The real difference is that a pipeline keeps a text transcript and inspectable stage boundaries, which matters for debugging, evaluation, and audit.

How much latency does the phone network add? Cisco's engineering team measures roughly 500 ms across the PSTN call path before AI processing. That fixed cost is why web demos mislead: the same stack that feels instant in a browser can land near 1.3 seconds on a real phone call.

How do I measure voice agent latency? Timestamp five events per turn (end of speech, final transcript, LLM first token, TTS first byte, audio out) and report P50, P95, and P99 for end-of-speech to first audio byte. Alert at P95 above 800 ms and P99 above 1,200 ms, and test with recordings that include real pauses and restarts, not scripted clean turns.

Where to go from here

A voice agent that answers within a second, waits politely through a caller's pause, and keeps an auditable transcript is an engineering system, not a single API key: telephony, streaming STT, a fast reasoning model, streaming TTS, and the measurement loop to keep the tail in check. If you want that built and run for you, our custom builds team designs and ships production voice agents on exactly this architecture, and can start from a working latency budget like the one above rather than a blank page.