How to handle real-time interruptions in your AI voice agent

How to handle real-time interruptions in your AI voice agent

Most AI voice agents perform great in the demo environment, but real life isn’t all quiet rooms, patient callers, and clean back-and-forths. Real life is messy, loud, frustrated, and interruptive. Your customers:

• Call from the car with a baby crying in the backseat

• Cut in halfway through an agent’s answer

• Repeat “uh-huh” in the middle of your agent’s sentence

Interruption after interruption. It’s chaotic, but it’s real. Your agent has to know when to talk, when to listen, when to talk over the caller's noises, and answer questions nobody has actually asked yet. And if your voice agent can’t handle the messiest encounters, your customers won’t trust them with the clean, simple ones either.

Over half of your customers (54%) want AI that sounds human, and nothing breaks that immersion like an AI agent who can’t tell the difference between a question and a cough. Fortunately, you can fine-tune these controls with your AI agents to help them handle every call with a bit more grace. Below, we’ll show you how AI voice agent interruption handling works, which settings control it in Twilio Conversation Relay, and what to tune when (not if) real calls get messy.

What is interruption handling in voice AI?

Interruption handling is how an AI voice agent detects that a caller has started talking, decides whether that sound is a real attempt to take a turn, and then stops, listens, or keeps going.

You'll also hear it called barge-in—this is a term borrowed from phone menus (IVRs) that let callers talk over the prompts instead of waiting them out. You know, like saying “talk to agent” before the IVR finishes giving you the 9 different options.

Humans handle interruptions without thinking. We’ve been doing it since we learned language. The rules of speech and turn-taking aren’t necessarily consistent or written out, but we just…get them. For example, we:

• Hear someone start a sentence

• Stop mid-word

• Unintentionally mumble or add ums and uhs

• Break written grammar rules 

• Know how much of our point landed

It’s natural, but a voice agent has to do each of those steps deliberately (in a few hundred milliseconds) over a phone line.

In Conversation Relay, that work is split between Twilio and your application. Conversation Relay:

1. Handles speech-to-text (STT), text-to-speech (TTS), and the live call audio

2. Connects to your app over a WebSocket

3. Detects the interruption and stops playback

4. Your app, and the LLM behind it, decides what the agent says next

3 types of interruptions your AI voice agent has to handle

Not every sound a caller makes deserves the same response. If they cough or a toddler screams in the background, you don’t want your AI pausing to ask, “What was that?” or “Can you repeat that?”

Treat everything as a legitimate interruption, and your agent turns twitchy, stopping every time someone breathes near the phone. However, if you ignore them all, you’ll get an annoying (sort of rude) agent. 

There’s a middleground here, but it takes a little tech and know-how:

1. Barge-in: The caller cuts the agent off

The agent starts explaining the return policy, the caller already knows it, and they jump in with "I just need the shipping label." 

A well-designed agent stops, listens, and answers the new request.

In Conversation Relay, the interruptible attribute controls which caller input can stop the agent's speech: 

• none

• dtmf (keypad presses)

• speech

• any

The default is any, so callers can interrupt out of the box.

2. Backchannels and overlapping speech

People make small noises while they listen. "Yeah." "Mm-hm." "Okay." Those sounds tell the speaker to keep going.

An agent that stops every time it hears "uh-huh" acts like it's walking on eggshells. Conversation Relay's ignoreBackchannel attribute filters out short phrases so they won't trigger an interruption. It's off by default, and Twilio supports backchannel detection across multiple languages, including English, Spanish, French, and German.

“ignoreBackchannel only applies while Hoot is talking. A caller saying "yeah" as a standalone turn after Hoot has finished is a legitimate prompt, and the agent will respond to it regardless of this setting.” — Dylan Frankcom, How to Handle Background Noise When Using Conversation Relay

3. Background noise that sounds like speech

Speech detection hears that loud vacuum or the coworker on another call. It’s picking it all up. If it decides the caller is talking, the agent will stop mid-sentence for nothing.

Noise causes a second, sneakier problem too. It garbles the transcript. The LLM doesn’t hear the call audio—it only sees the text. 

That means a bad transcript will always lead to a bad answer.

Dylan Frankcom tested this theory on speakerphone at arm's length with a vacuum (~70 to 75 dB), a power drill (~85 to 90 dB), and both running at once. The default Nova-3 setup produced two transcription failures under combined noise ("loyalty points" came through as "loyalty forms"), but a tuned Nova-3 setup and Deepgram Flux both finished with zero.

You can't clean up caller audio with preprocessing libraries in a Conversation Relay pipeline because your server never receives the raw audio. Twilio routes it straight to the transcription provider. Noise fixes happen in the speech model and the settings.

How turn detection decides who's talking

Interruption decisions start with turn detection. Your agent has to figure out when a caller has started talking and when they've finished. Get it wrong in one direction and the agent cuts people off. Get it wrong in the other and callers sit through awkward silence.

The traditional approach pairs voice activity detection with a silence timer. The system listens for speech, waits for a set stretch of silence, then calls the turn complete. You can probably guess where this fails, though:

• A mid-thought pause looks like the end of a turn

• Loud noises look like speech

• Background noise may never allow full silence

In Conversation Relay, the speechTimeout attribute sets how long to wait after speech ends before sending the final prompt to your app, anywhere from 600 to 5,000 milliseconds.

Newer models judge the end of a turn by what the caller is saying (with silence as just one signal). Conversation Relay now supports Deepgram Flux, our partner's model that fuses transcription and turn detection into one step. According to Deepgram, this:

• Removes the race conditions that cause agents to cut callers off

• Trims response latency by 200 to 600 milliseconds

• Reduces false interruptions by roughly 30% compared to Nova-3

Flux has a dial you can adjust: eotThreshold. It’s the confidence the model needs before it finalizes a turn. It accepts values from 0.5 to 0.9 and defaults to 0.8. With Flux, speechTimeout becomes a maximum silence cap, forcing the turn to end if a caller goes quiet for long enough.

Turn detection when callers switch languages

Multilingual callers make this process a little less straightforward, but there’s a solution. Conversation Relay supports automatic language detection when you set the speech-to-text language to multi, using Deepgram for transcription and ElevenLabs for speech. Each prompt your app receives includes the detected language, and your app can send a switch language message to change languages mid-call. 

Your interruption step depends on what LLMs think was said

This is where demos don’t reproduce real-world use cases. 

Your LLM writes a full answer, and your app streams it to Conversation Relay token by token. The caller interrupts after the first sentence. The audio stops right on cue. But your conversation history still contains the entire answer, so on the next turn the LLM believes the caller heard all of it. 

It might say "as I mentioned, the fee is waived" when the caller never heard a word about fees. 

Woof.

Fortunately, Conversation Relay gives your app what it needs to fix this. When a caller interrupts, it sends an interrupt message over the WebSocket with the text that played before the cutoff and how long the agent had been speaking.

Conversation Relay detects the interruption and pauses playback. Managing conversation history stays with your app with just a few lines of extra code. Find the most recent assistant message that contains utteranceUntilInterrupt, then cut it off at that point so the history matches what the caller heard. We have walkthroughs for this with both OpenAI and Anthropic models.

How to tune interruption handling in Conversation Relay

Most interruption problems come down to a handful of TwiML attributes. Start here before anyone writes new code.

Learn more about each of these controls:

• TwiML™ Voice: <ConversationRelay>

• Getting and sending WebSocket messages

• How to Handle Background Noise When Using Conversation Relay

Now, there’s no one-size-fits-all best settings. That’s why they’re adjustable. Every setting comes with a tradeoff. Lower sensitivity means a caller who really wants to interrupt has to speak a bit more deliberately. For most support calls, that's a fair trade for an agent that doesn't stop every time a power tool starts up nearby.

Latency decides if your interruptions feel natural (or weird)

Latency and interruptions are two sides of the same problem. When an agent takes too long to respond, callers fill the silence, and now your agent is being interrupted before it's said a word.

Conversation Relay is built for live conversation, with a median latency under 0.5 seconds and under 0.725 seconds at the 95th percentile.* Your app controls the rest of the delay:

• Stream tokens as they arrive: Send text tokens to Conversation Relay as soon as your LLM produces them instead of waiting for the full response.

• Start thinking before the caller finishes: With Flux, the partialPrompts attribute sends unfinalized prompts and early end-of-turn signals to your app, so it can start work before the turn is confirmed. Plan to throw that work away if the caller keeps talking.

*Based on internal benchmarks with Conversation Relay (p50 491 ms, p95 713 ms) using different models. Results may vary.

How to tell if interruption handling is working

Callers don’t really leave complaints using the same terminology we’re accustomed to (if they leave a complaint at all). Nope, they just hang up, repeat themselves, or ask for a human. That makes the problem easy to miss unless you intentionally look for it.

A few signals to watch:

• Interruptions per call: Every interrupt message is a data point. A jump after a configuration change usually means sensitivity is set too high for your callers' environments.

• Interruptions in the first second: The durationUntilInterruptMs value shows how long the agent had been speaking. A cluster of very early interruptions can point to noise or backchannels tripping detection.

• Repeat requests: When callers say "what?" or ask the same question twice, look for cut-off responses or garbled transcripts.

• Early hang-ups and escalations: These are the lagging indicators. By the time they move, callers are already upset.

Conversation Relay helps you capture these signals. The events attribute can notify your app when the agent or the caller starts speaking, and report what's been played over TTS. The intelligenceService attribute sends transcripts to Conversation Intelligence and runs Language Operators for virtual agent observability. Conversation Intelligence also detects shifts in customer sentiment and tone, which is handy for spotting the calls where interruptions didn’t quite go according to plan.

When to hand the call to a human

Your AI agent can only do so much. Some calls don’t need better AI—they just need a human. A caller in a loud train station who's repeated their account number three times doesn't want a fourth attempt. They want a person.

When your app decides it's time, it can end the Conversation Relay session with a handoff reason. Twilio then calls your <Connect> action URL with that HandoffData, like "The caller requested to talk to a real person," so your call flow can route to a live agent.

The handoff only helps if the human doesn't start from zero, though. Conversation Orchestrator passes the full context, history, and intent to the agent who picks up, so the caller doesn't have to repeat the story they've already told twice.

Put your agent through a real call

You don’t need a rebuild. You just need to tweak a few TwiML attributes and add a few lines of code. That’s a short list of changes, and testing them is the fastest way to see whether your agent handles interruptions like a person or like a vending machine.

Here's how to get started:

• Hear it first: Try the Conversation Relay demo and interrupt it on purpose. Talk over it, say "uh-huh," and turn on the faucet.

• Build your own: Sign up for a free Twilio account (no credit card required), then follow the Conversation Relay onboarding guide to connect your first voice agent.

• Borrow a tested setup: Start from the configuration in Twilio's background noise tutorial, which comes in Python, JavaScript, C#, and PHP.

Then call your agent from the loudest room you can find. If it keeps up there, congratulations—it's ready for your customers.

FAQ

What is barge-in in voice AI?

Barge-in is when a caller starts talking while an AI voice agent is still speaking. Twilio Conversation Relay handles it with the interruptible attribute, which stops the agent's speech when caller audio or keypad input comes in.

How does Twilio Conversation Relay handle interruptions?

Twilio Conversation Relay detects caller speech during playback, stops text-to-speech, and sends your app an interrupt message with the exact text played before the cutoff. Your app then trims the LLM's history to match.

How do I stop background noise from interrupting my AI voice agent?

Twilio's testing found Deepgram Flux handled heavy noise on its default settings. On Nova-3, set interruptSensitivity to low and ignoreBackchannel to true. Both setups had zero transcription failures with a vacuum and drill running.

Can I stop callers from interrupting specific messages?

Twilio Conversation Relay lets you set interruptible to false on individual text messages. Required disclosures and confirmations play in full, while the rest of the call stays interruptible.

Does interrupting a voice agent break the LLM's context?

Twilio Conversation Relay stops the audio, but your app manages conversation history. Use the utteranceUntilInterrupt value to trim the assistant's last message so the LLM only remembers what the caller heard.

Can an AI voice agent handle callers who switch languages mid-call?

Twilio Conversation Relay supports automatic language detection when speech-to-text is set to multi, using Deepgram for transcription and ElevenLabs for speech. Your app can also switch languages mid-session.