Update configuration during a call

During a call, you can:

• Change a language's settings: Update the speech model, hints, transcription provider, voice, TTS provider, and text normalization for a language.
• Change session settings: Update interruptible, preemptible, dtmfDetection, reportInputDuringAgentSpeech, partialPrompts, speechTimeout, echoPromptFiltering, interruptSensitivity, ignoreBackchannel, profanityFilter, and eotThreshold.
• Switch the active language: Change the TTS language and the transcription language independently or together. For example, you can speak one language while recognizing another.
• Add a language that your TwiML didn't declare: Declare the language and switch to it in a single message.
• Speak one reply in another language: Tag the text tokens of a reply with lang instead of switching the session language.
• Receive configuration events: Subscribe to the config-updated event to learn which configuration is in effect.

An update changes only the settings that you name. Every setting that you leave out keeps its current value. Sending a value that's already in effect has no effect, so you can re-send settings without tracking what changed. Conversation Relay still validates the whole message, so leave out settings that you can't update or that your account can't use.

Settings that define the session, rather than how it behaves, stay fixed for the session. Conversation Relay rejects a config message that includes any setting not listed in What you can update, such as:

• The WebSocket url
• The welcomeGreeting and welcomeGreetingInterruptible attributes
• The events subscription
• The intelligenceService and conversationConfiguration bindings
• The applicationReconnect and partialPromptOnHangup attributes

To change any of these settings, start a new session.

Send updates with a config message. The config message accepts two fields. Include at least one setting in either field:

• language: The settings of one language, using the same names as the <Language> element attributes. Pass an array to edit several languages at once. Set code to name the language to edit. Omit code, or send current, to edit the language that's in use.
• session: The session settings, using the same names as the <ConversationRelay> attributes.

1
{
2
  "type": "config",
3
  "language": {
4
    "code": "en-US",
5
    "speechModel": "nova-3-general",
6
    "hints": "invoice,autopay"
7
  },
8
  "session": {
9
    "dtmfDetection": true,
10
    "interruptible": "speech"
11
  }
12
}

To switch languages, set ttsLanguage, transcriptionLanguage, or both in session. You can send either one on its own.

1
{
2
  "type": "config",
3
  "session": {
4
    "ttsLanguage": "es-ES",
5
    "transcriptionLanguage": "es-ES"
6
  }
7
}

To add a language that your TwiML didn't declare, include its settings in the same message that switches to it. Conversation Relay puts the settings in place before it applies the switch.

1
{
2
  "type": "config",
3
  "language": {
4
    "code": "de-DE",
5
    "ttsProvider": "ElevenLabs",
6
    "voice": "FTNCalFNG5bRnkkaP5Ug",
7
    "transcriptionProvider": "Deepgram",
8
    "speechModel": "nova-3-general",
9
    "hints": "Rechnung,Guthaben"
10
  },
11
  "session": {
12
    "ttsLanguage": "de-DE",
13
    "transcriptionLanguage": "de-DE"
14
  }
15
}

Provide each part of a new language in full: ttsProvider and voice for text-to-speech, and transcriptionProvider and speechModel for speech recognition. Conversation Relay doesn't fill in default settings for a language that you add during the call, even for a language that has defaults at session start. You can add only one part, for example only text-to-speech, and switch only that part to the new language.

Conversation Relay rejects the update with a TTS for '<language>' is not configured or Transcription for '<language>' is not configured error when:

• You switch to a language that the session doesn't have, without its settings.
• You give only one setting of a part that the language doesn't have yet, for example voice without ttsProvider.
• You add a language with only other settings, such as hints.

The call continues with its current configuration.

For the updates that applications send most often, you can use shorthand messages. Each shorthand message is applied like the equivalent config message.

Use the hints message to set speech recognition hints for the language that Conversation Relay is recognizing:

1
{
2
  "type": "hints",
3
  "hints": "invoice,balance,autopay,routing number"
4
}

Use the language message to switch languages. You can include hints, which apply to the transcriptionLanguage, so a message with hints must also include transcriptionLanguage. The message also accepts settings for the target languages, such as voice and speechModel, and can add a language when it gives each part in full. See Add a language and switch to it.

1
{
2
  "type": "language",
3
  "ttsLanguage": "es-ES",
4
  "transcriptionLanguage": "es-ES",
5
  "hints": "factura,saldo,pago"
6
}

A language switch changes the session, so every reply after the switch is spoken in the new language. To speak only one reply in another language, tag the text tokens of that reply with lang instead. For example, use lang for a phrase for a caller who switched languages briefly, or for a name that you read in its own language.

1
{
2
  "type": "text",
3
  "token": "Hola, ",
4
  "lang": "es-ES",
5
  "last": false
6
}

1
{
2
  "type": "text",
3
  "token": "¿en qué puedo ayudarte?",
4
  "lang": "es-ES",
5
  "last": true
6
}

Keep the following in mind when you tag tokens:

• The tag applies to that reply only. Conversation Relay speaks the reply with the voice and TTS provider of the tagged language. Your session's TTS language doesn't change, so Conversation Relay speaks the next tokens that you send without lang in the session's TTS language. Speech recognition isn't affected.
• Tag the first token of the reply. The lang on a reply's first token sets the language for the whole reply, up to the token with last set to true. Conversation Relay ignores lang on later tokens of the same reply, so tagging every token is harmless but not required.
• The language must be configured. Declare the language in TwiML, or add it with a config message. If the session doesn't have TTS settings for the language, Conversation Relay doesn't speak the tokens and sends you error 64106.
• Replies stay in order. Some TTS providers need a connection of their own. The first reply that uses one of these providers might start slightly late while the connection opens. Replies that you send after it wait their turn, and Conversation Relay never reorders them.

Conversation Relay applies each change at the earliest point where it can do so without losing data. Which point depends on the setting that you change.

Keep the following in mind:

• A language switch doesn't wait. It applies as soon as it validates, together with everything else in the same message and any update still waiting for the end of the turn. If the caller is speaking, recognition restarts and the caller's speech in progress is discarded.
• A reply tagged with lang is spoken in that language from its first token. Conversation Relay doesn't cut short the reply before it.
• A synthesis update that arrives mid-reply splits the reply: text that you sent before the update keeps the old settings, and text after it uses the new ones.
• If you send several recognition updates during one caller turn, Conversation Relay applies them together when the turn ends, with the last value of each setting, in a single stream re-establishment.

For most applications, send an update when the conversation turns, and expect the change on the next exchange.

Successful updates are silent by default. Conversation Relay doesn't acknowledge an update. You observe the change in the conversation, as you do with the language message.

To find out which configuration is in effect, subscribe to the config-updated event in your TwiML:

1
<Response>
2
  <Connect>
3
    <ConversationRelay url="wss://example.com/socket" events="config-updated" />
4
  </Connect>
5
</Response>

When you subscribe, Conversation Relay sends an info message named configUpdated whenever the configuration in effect changes. The value field has the same shape as a config message, so you can read it in the same way. Before you send it back, remove settings that your account can't update.

1
{
2
  "type": "info",
3
  "name": "configUpdated",
4
  "value": {
5
    "language": [
6
      {
7
        "code": "es-ES",
8
        "voice": "Lucia-Neural"
9
      }
10
    ]
11
  }
12
}

Conversation Relay reports configuration as follows:

• At session start: You receive the full configuration that the call is running with, including defaults that you never set. It arrives as three events, each when its part goes live: the session settings, the TTS settings of the session's TTS language, and the speech recognition settings of the session's transcription language.
• After each change: You receive only what changed, at the moment it takes effect, not when your message arrives. Session, TTS, and recognition settings arrive as separate events, and each language entry names its code. A language switch reports ttsLanguage and transcriptionLanguage under session.
• For a reply in another language: When you tag a reply with lang, Conversation Relay reports the TTS settings that the reply uses. When the next reply returns to the session's TTS language, Conversation Relay reports that change in the same way. These events have no session part, because the session's TTS language doesn't change.
• When nothing changes: Conversation Relay sends no event for an update that's already in effect, or for a change to a language that the call isn't using.
• Value format: Values are strings, in the same vocabulary that you use in TwiML, such as "true" or "elevenlabs".

Treat the event as a statement that the configuration is now a specific state, not as a receipt for a message that you sent. Conversation Relay reports several updates that it applies together as one change, and it reports a reply tagged with lang even though you sent no update.

Updates are atomic. If any value in a config message is invalid, Conversation Relay rejects the entire message, including the entries that were valid, and sends you an error. The call continues with the configuration it had before. Retry the whole message instead of assuming that part of it applied. A rejected message changes nothing. If a valid update then fails to apply, for example when the recognition stream can't restart, you receive an error after the update has been accepted.

Most rejections arrive as an error message whose description explains the problem. They don't carry an error code and don't end the call. Structural problems return error 64107.

Ten consecutive invalid messages close the connection. Conversation Relay also rate limits update messages. Don't re-send configuration in a tight loop.

This guide teaches the basics required for the following use cases:

You can use this guide to change an AI agent's behavior as a conversation develops. For example, you can switch to the caller's language when you detect it, add domain-specific hints when the topic changes, or turn on DTMF detection for a payment step.

To learn more advanced features that you can use with AI agents, see AI agents.

You can use this guide to adjust recognition settings, such as the speech model and hints, while a call is in progress so that transcription accuracy follows the content of the call.

To learn more advanced features that you can use with AI/ML transcription, see Voice AI/ML transcription.

After following this guide, you can change language, voice, and session settings during a Conversation Relay call. You can confirm that it works by sending a config message, and then verifying that your application receives a configUpdated info message when the change takes effect.

Explore the following guides to build on what you've learned in this guide:

• Getting and sending WebSocket messages: Review the reference for every message that you can send and receive.
• Picking a voice: Choose the TTS provider and voice to use for each language.
• Best practices for Conversation Relay: Learn how to handle connection drops, language switching, and false interruptions.

Find reference documentation for the topics on this page.

• <ConversationRelay> TwiML noun
• TwiML™ Voice: <ConversationRelay>
• Conversation Relay WebSocket messages