Turn detection for Conversation Relay

A voice AI agent needs to know when the caller has finished speaking. If the agent responds too early, it interrupts the caller. If it waits too long, the conversation feels slow.

By default, Conversation Relay relies on the speech-to-text (STT) provider's silence handling to end the caller's turn. A silence threshold can't tell a caller who has finished from one who is pausing to think or using filler words, which leads to false interruptions.

Turn Detection replaces that silence-based decision with a semantics-aware machine learning model built by Twilio. The model evaluates the caller's audio, not the transcript, in real time to tell whether the caller has completed their turn or is pausing mid-sentence. It runs in Twilio's media stack alongside STT, so it works with all supported STT providers and speech models except Deepgram Flux. For more on Flux, see Limitations.

When the model decides the turn is complete, Conversation Relay sends the prompt message with last set to true, instead of waiting for an STT silence timer to expire. Turn Detection only affects when the turn ends. It doesn't change how Conversation Relay detects interruptions of the agent's speech.

Turn Detection is most useful when callers often pause in the middle of a thought. For example:

• Reading out long values. Callers who recite confirmation codes, tracking numbers, or phone numbers tend to pause between groups of digits.
• Hesitant or thoughtful conversations. In technical support or claims calls, callers often stop to collect their thoughts or use filler words such as "um" and "ah".
• Booking and reservations. A caller might say "I'd like a table for... four people on Friday night." An agent that replies after the first pause breaks the flow of the conversation.

Set the turnDetection attribute to true on the <ConversationRelay> noun:

1
<?xml version="1.0" encoding="UTF-8"?>
2
<Response>
3
  <Connect>
4
    <ConversationRelay
5
      url="wss://YOUR_SERVER_URL"
6
      welcomeGreeting="Hi! How can I help you today?"
7
      turnDetection="true"
8
    />
9
  </Connect>
10
</Response>

No changes to your WebSocket server are required. Your application continues to receive prompt messages and treats last: true as the end of the caller's turn.

1
{
2
  "type": "prompt",
3
  "voicePrompt": "I would like to book a table for Friday night.",
4
  "lang": "en-US",
5
  "last": true
6
}

If your application needs more detail about the caller's turn, set the includeInPrompt attribute to turn-state. Conversation Relay then adds a turnState field to the prompt messages it sends. To receive the start and interim states, also set partialPrompts to true.

1
<?xml version="1.0" encoding="UTF-8"?>
2
<Response>
3
  <Connect>
4
    <ConversationRelay
5
      url="wss://YOUR_SERVER_URL"
6
      turnDetection="true"
7
      includeInPrompt="turn-state"
8
      partialPrompts="true"
9
    />
10
  </Connect>
11
</Response>

With this configuration, the prompt message that ends the caller's turn looks like this:

1
{
2
  "type": "prompt",
3
  "voicePrompt": "I would like to book a table for Friday night.",
4
  "lang": "en-US",
5
  "last": true,
6
  "turnState": "end"
7
}

The turnState field can be start, interim, or end. For what each value means and when Conversation Relay sends it, see the prompt message with turn state reference.

• speechTimeout: Sets the longest silence that Conversation Relay allows before it ends the caller's turn, even if the model hasn't signaled that the turn is complete. If you don't set speechTimeout, Conversation Relay uses 1500 milliseconds. Valid values are integers from 600 to 5000.
• eotThreshold: Sets the confidence required to end a turn. Allowed values are 0.5 to 0.9. The default is 0.8.

• Deepgram Flux: Turn Detection works with all supported STT providers and speech models except Deepgram Flux, which has its own turn detection model built in. If you enable turnDetection and also set transcriptionProvider to Deepgram and speechModel to flux, Conversation Relay bypasses Twilio Turn Detection and uses the Flux turn detection model. You can still use includeInPrompt="turn-state" with Flux. To tune Flux turn detection, see the eotThreshold and speechTimeout attributes in the <ConversationRelay> reference.

Explore the following guides to build on what you've learned in this guide:

• Best practices for optimizing your Conversation Relay app: Learn how to handle caller silence, prevent false interruptions, and stream text tokens.
• Update configuration during a call: Change language, voice, and session settings over your WebSocket connection without ending the call.

Find reference documentation for the topics on this page.

• TwiML™ Voice: <ConversationRelay>: Reference documentation for the turnDetection and includeInPrompt attributes.
• Conversation Relay WebSocket Messages: Reference documentation for the prompt message and its turnState field.