Voice Agent Barge-In: How Interruption Handling Really Works

When a caller talks over your voice agent and the agent keeps reading its script for three more seconds, the call is effectively over. Barge-in, the ability for a human to interrupt an AI voice agent mid-sentence and be heard, is the single feature that separates a voice agent that feels like a conversation from one that feels like a phone tree with better pronunciation. Getting it right requires four subsystems working together: turn detection, an interruption cascade that stops audio fast, context trimming so the model knows what was actually said, and filtering that ignores backchannels and noise. This guide walks through each one with real defaults from LiveKit, Twilio, Deepgram, OpenAI, Pipecat, and Amazon Nova, and ends with a test plan you can run before your callers find the bugs for you.

Why barge-in is three problems, not one

Teams usually say "interruption handling" as if it were one toggle. It is three separate problems that need different solutions, as Twilio's engineering team puts it:

• Barge-in: the caller deliberately cuts the agent off. "Wait, I said the 14th, not the 4th." This must stop the agent immediately and be treated as new input.
• Backchannels: short listener signals like "uh-huh," "right," "okay." These must NOT stop the agent. If every "mm-hm" halts playback, the agent stutters through every sentence.
• Background noise: a vacuum cleaner, a truck horn, a TV. This should be ignored entirely, not transcribed as speech and not treated as an interruption.

A system that treats all three the same way fails in both directions: it either ignores real corrections (callers repeat themselves, get frustrated, hang up) or it flinches at every noise (the agent can never finish a sentence). Hamming, a voice-agent testing platform, describes interruption handling as a policy and instrumentation layer rather than a feature toggle, and their taxonomy adds DTMF tones, silence timeouts, and safety escalations to the list of inputs that all arrive through the same "someone made a sound" channel.

How the agent decides you are talking

Before anything can be interrupted, the system has to decide that a turn has started or ended. There are four mainstream approaches, and the LiveKit turn detection documentation lays out the trade-offs clearly:

• VAD only. A voice activity detector fires when energy in the audio looks like speech. Lowest latency, works in any language, but it cannot tell "uh-huh" from "stop," and it will fire on a cough.
• STT endpointing. The speech-to-text engine emits end-of-turn events. Deepgram's Flux model is the current reference here: it fuses transcription and turn detection in one model and emits a state machine of StartOfTurn, EagerEndOfTurn, TurnResumed, and EndOfTurn events. Every StartOfTurn is guaranteed to carry a non-empty transcript, which makes it a more reliable barge-in trigger than a raw external VAD.
• A turn detector model. A small model predicts end-of-turn from both meaning and acoustics, so "I need to change my flight to... " (trailing off) is handled differently from a complete sentence. This is LiveKit's default and recommended mode.
• Realtime LLM built-in detection. Speech-to-speech models like OpenAI Realtime and Amazon Nova Sonic do their own server-side turn detection. With these, your framework's interruption settings are mostly ignored; you tune the model's own VAD instead.

OpenAI's Realtime API offers two VAD modes worth knowing by name: server_vad, the default, which is a classic energy-and-silence detector with a threshold, prefix padding, and silence duration, and semantic_vad, where the model itself decides the utterance is complete based on the words, with an eagerness setting from low to high. The OpenAI VAD guide documents both. Semantic VAD is better at not cutting off slow, thoughtful speakers; server VAD is more predictable to tune.

Deepgram's reported numbers give a sense of what better endpointing buys: Flux cuts agent response latency by 200 to 600 milliseconds and reduces false interruptions by about 30 percent compared with pipeline approaches. Those are vendor numbers, so treat them as directional, but the direction matches what the architecture predicts.

The interruption cascade: what must happen in the first 200 milliseconds

Once the detector says "the human is talking," a chain of things has to happen fast. The Pipecat documentation on interruptions describes the cascade precisely, because Pipecat implements it as a system frame that jumps the queue:

1. Cancel the LLM completion mid-stream. The agent was probably still generating tokens for a sentence the caller will never hear.
2. Clear the TTS buffer. Any synthesized audio not yet sent to the transport is discarded.
3. Flush the playback queue. This is the step teams forget. Audio is generated faster than it plays, so at any moment there is queued audio sitting in the output buffer. If you stop generation but not playback, the caller hears another second or two of the old response and concludes the agent ignored them.
4. Start processing the new input as the next turn.

Amazon's Nova 2 Sonic barge-in documentation describes exactly the same contract from the other side: the model stops generating server-side, preserves the full conversation context, and sends an interruption signal; the client must detect that signal, stop playback, and clear its audio queue. Whichever vendor you use, the queue flush is on you.

Done right, the bot goes silent within roughly one audio write of the interruption. Done wrong, you get the classic complaint: "I told it to stop and it kept talking."

The context problem: trim what the caller never heard

Stopping the audio is only half the job. The other half is the conversation history you send to the LLM on the next turn.

Say the agent's planned response was: "Your appointment is confirmed for Tuesday the 14th at 3 pm, and please arrive ten minutes early to complete registration." The caller barges in after hearing only "Your appointment is confirmed for Tuesday the" and says "wait, I said the 4th." If your LLM history still contains the full sentence including "the 14th," the model will reason about a date the caller never heard confirmed. The correction "I said the 4th" becomes ambiguous, and the agent may confidently rebook the wrong day.

Both major frameworks handle this the same way. Twilio's Conversation Relay sends an interrupt message containing the exact text played before the cutoff, utteranceUntilInterrupt, and expects your app to trim the LLM history to match. Pipecat syncs text frames with audio playback and, on interruption, commits only the words actually spoken as the assistant's message. LiveKit exposes the same idea through a false-interruption timeout and resume flag.

The rule to hold your implementation to: the model's memory of what it said must match what the caller's ears heard. Anything else is a silent state bug that shows up later as inexplicably wrong bookings, quotes, or amounts.

False interruptions: backchannels and noise

The opposite failure is an agent that flinches at everything. There are four knobs, roughly in order of sophistication:

• Minimum duration. LiveKit's default is min_duration of 0.5 seconds of continuous speech before an interruption registers, per its turn-taking tuning guide. A cough or a click rarely lasts that long.
• Minimum words. Requiring two or three transcribed words before treating input as a turn filters most "uh-huh" backchannels. LiveKit has min_words, Pipecat has MinWordsUserTurnStartStrategy, both for exactly this.
• Semantic backchannel filtering. Twilio's ignoreBackchannel attribute suppresses short listener phrases across multiple languages. LiveKit's adaptive interruption mode uses an audio model to distinguish a genuine interruption from a backchannel.
• Noise-aware models. Dedicated interruption-prediction models such as Krisp VIVA, which Pipecat integrates, classify whether incoming audio is an interruption, a backchannel, or noise before any state changes.

The stakes are measurable. In a Twilio noise test, a tester ran a vacuum at 70 to 75 dB and a power drill at 85 to 90 dB next to a speakerphone on a live call. A default Deepgram Nova-3 configuration produced two transcription failures under combined noise, turning "loyalty points" into "loyalty forms." A tuned Nova-3 setup and Deepgram Flux produced zero. That is the difference between an agent that books the right thing and one that hallucinates a form request because someone cleaned the office.

The tuning knobs that matter, with real defaults

If you are configuring a stack this week, these are the documented starting points:

• LiveKit: endpointing min_delay 0.5s and max_delay 3.0s; interruption min_duration 0.5s; false_interruption_timeout 2.0s with resume on by default. If the agent cuts callers off mid-thought, raise endpointing delay; if it talks over people, lower min_duration or enable adaptive mode.
• OpenAI Realtime on telephony: LiveKit recommends server_vad with threshold 0.7, prefix_padding_ms 300, silence_duration_ms 400. A higher threshold survives noisy phone lines better than the API defaults.
• Twilio Conversation Relay: interruptible accepts none, dtmf, speech, or any, and defaults to any. speechTimeout runs 600 to 5000 ms. Set interruptible to none for individual messages that must play in full, such as legal disclosures.
• Amazon Nova Sonic: turn-taking sensitivity maps to waits of roughly 1.5, 1.75, or 2.0 seconds of silence before the model responds.

One non-obvious trap, flagged in the Hamming runbook: some platforms separate "stop playback" from "keep listening." Twilio, for example, has interruptible for stopping audio and reportInputDuringAgentSpeech for passing caller input to your app while the agent talks. You can build an agent that hears corrections without stopping, or stops without hearing. Decide which behavior each turn type needs; they are not the same decision.

A worked example: what eager end-of-turn costs

Deepgram Flux has an eager mode where it emits an EagerEndOfTurn event 150 to 250 ms before the final EndOfTurn, so your LLM can start generating early. The documented cost: 50 to 70 percent more LLM calls, because some eager turns get resumed and the speculative generation is discarded.

Run the numbers for a deployment handling 5,000 calls a month at 15 agent turns per call. That is 75,000 turns. Eager mode adds roughly 38,000 to 52,000 speculative LLM calls. On a mid-tier hosted model at about $1.50 per million input tokens, with a 600-token prompt per call, that is $34 to $47 a month in discarded speculation, buying you a fifth of a second on every turn. For a booking line, that is usually worth it. For an internal hotline doing 200 calls a month, it is noise. The point is not the exact figure; it is that every latency knob has a computable price, and you should compute it before turning the knob.

How to test interruption handling before your callers do

Hamming's production runbook, built from analysis across more than 10,000 voice agents, reduces to a procedure you can run in a week:

1. Freeze a test set. Record 20 to 30 scenarios: true corrections mid-sentence, pure backchannels, background noise during playback, long entity strings (account numbers read digit by digit), silence timeouts, and escalation phrases like "get me a human."
2. Log every interruption as a structured event. Capture input type, system decision, whether playback was already committed to the caller, and whether the turn completed. If you cannot answer "why did the agent stop there?" from logs, you have no evidence layer.
3. Change one setting at a time. Never tune endpointing delay and VAD threshold in the same release; you will not know which one fixed or broke the behavior.
4. Track the two failure rates separately. False interruption rate (agent flinched at noise or backchannel) and missed interruption rate (agent ignored a real correction) move in opposite directions when you tune. Optimizing one number alone is how teams ship an agent that either stutters or steamrolls.
5. Review the top 20 interrupted production calls after every release. Not a sample of all calls; specifically the interrupted ones, because that is where the policy lives.

If you are under roughly 50 production calls a week, Hamming's own advice is to keep it simple: conservative defaults, five to ten regression tests, and manual review. The full metrics program pays for itself at volume, not before.

Common mistakes and when barge-in is the wrong choice

The mistakes that recur across stacks:

• Stopping generation but not the playback queue, so the agent "keeps talking" after a successful interruption.
• Never trimming LLM history, so the model reasons about words the caller never heard.
• One global interruptible setting, when legal disclosures and confirmation readbacks should be non-interruptible while everything else should not be.
• Tuning with clean studio audio. Telephony audio is 8 kHz, compressed, and noisy; a threshold that works on your laptop microphone will misfire on a Safaricom call from a matatu.
• Enabling interruptions with a slow pipeline. If your stack takes two seconds to produce a new response after an interruption, callers interrupt, hear silence, and hang up. Fix latency first.

And sometimes barge-in is simply wrong. Regulated readbacks (consent language, card numbers read for confirmation) should play in full with interruptible set to none. Outbound notification calls under a few hundred a week are better served by a well-designed DTMF flow than a fully interruptible conversational agent. If your callers' most common "interruption" is actually "speak to a human," spend the budget on a fast, reliable escalation path rather than on turn-detection subtlety.

Frequently asked questions

What is barge-in in a voice AI agent? Barge-in is the ability for a caller to interrupt the agent while it is speaking and have the agent stop, listen, and respond to the new input. It requires turn detection, immediate cancellation of the response pipeline, flushing of queued audio, and trimming of conversation history to what was actually played.

How do I stop my voice agent from interrupting callers? Raise the endpointing delay so short pauses do not end the caller's turn, require a minimum interruption duration or word count, and enable backchannel filtering. LiveKit's defaults are an endpointing min_delay of 0.5 seconds and an interruption min_duration of 0.5 seconds; both are documented tuning knobs.

Why does my agent keep talking after a caller interrupts? Almost always because queued audio was not flushed. Audio is synthesized faster than it plays, so there is a playback buffer; cancelling the LLM and TTS does not empty it. The output transport must be told to discard unplayed audio, which is the client's responsibility on every major platform.

What is the difference between VAD and semantic turn detection? VAD detects speech from audio energy and silence, so it fires on any sufficiently loud sound. Semantic or model-based turn detection uses the words and acoustic cues to judge whether the utterance is complete, which handles trailing pauses and backchannels far better at the cost of some latency.

Should legal disclosures be interruptible? Generally no. Platforms like Twilio Conversation Relay let you set interruptible to none per message, so consent language and regulated readbacks play in full while the rest of the conversation stays interruptible.

Does barge-in work with speech-to-speech models like OpenAI Realtime? Yes, but the model handles turn detection server-side and your framework's interruption settings are mostly ignored. You tune the model's own VAD parameters, such as server_vad threshold and silence duration, instead.

Where to go from here

Barge-in quality is where voice agent projects are won or lost, and it is fixable: pick a turn detection strategy, implement the full interruption cascade including the queue flush, trim context to what was played, and test both failure directions with real telephony audio. If you want a broader view of where voice agents fit in a business, our complete guide to AI voice agents covers the architecture and economics, and our breakdown of what a voice AI agent costs to run in Kenya prices the telephony and model stack. For teams building agents that must survive production beyond voice, our guide to error handling and retry strategies covers the resilience patterns that pair with the testing procedure above. If you would rather have engineers build and tune this for you, our custom builds team designs and ships production voice agents, and our AI employees service runs always-on agents as a managed engagement.