Best practices for Conversation Relay

If your application uses an LLM API then you likely have the option of receiving text tokens streamed to you in chunks from the LLM provider. Most applications should stream these text tokens to Conversation Relay as soon they become available instead of waiting for the LLM to generate a complete response. This will allow Conversation Relay to begin speaking the response to your caller sooner.

Alternatively, you can wait for the LLM to finish generating the entire response before sending it to Conversation Relay. This approach can introduce significant latency but it might offer a smoother and more consistent pace of speaking depending on your application and LLM. Test both approaches to see which works best for your use case.

It's important to try different speech-to-text (STT) providers and models to find the best fit for your application's needs. Google and Deepgram each offer unique strengths for different scenarios, such as clean versus noisy audio environments. You can configure your STT provider and model using the transcriptionProvider and speechModel attributes in the <ConversationRelay> noun in TwiML.

Proper text normalization is important for delivering clear and natural spoken responses. This is especially important when using ElevenLabs voices which may have difficulty with certain formats. Consider the following guidelines:

• Numbers and units: Write numbers as words for improved pronunciation. For example, write "twenty dollars and fifty cents" instead of "$20.50".
• Dates: Spell out dates completely (for example, "March twenty-eighth, two thousand twenty-five" instead of "03/28/2025").
• Email addresses: Replace or spell out special characters. For instance, write "user at example dot com" rather than "user@example.com" since the "@" sign may be mispronounced.
• Names: Use consistent formatting for names throughout the call to avoid variations (for example, always use "Anna" rather than alternating between "A-nna" and "Ah-nna").
• Abbreviations: Spell out abbreviations that should be pronounced fully (for example, "Doctor" instead of "Dr.").
• Special characters: Replace special characters with their spoken equivalents (for example, "percent" instead of "%").
• Punctuation: Use appropriate punctuation to create natural pauses and intonation.
• Acronyms and initials: Insert spaces between letters if they must be spelled out (for example, "H T T P" for letter-by-letter pronunciation).

If you use the ElevenLabs TTS provider then you can also use the elevenlabsTextNormalization TwiML attribute to help with text normalization. Setting this attribute to on will tell ElevenLabs to take an extra step to normalize text for you. Set the attribute to off for more control over the text normalization process and lower latency.

For detailed text normalization guidelines, refer to ElevenLabs' text normalization best practices.

Monitor for error messages from the Conversation Relay WebSocket connection. These messages can indicate a problem with the message that you sent us or an issue with the connection. Implement appropriate error handling in your application to manage these situations gracefully.

WebSocket connections between your application and Conversation Relay can drop because of autoscaling events, container redeployments, or load balancer recycling.

To prevent dropped calls during infrastructure outages, set the applicationReconnect attribute to true in your <ConversationRelay> TwiML.

If a connection drops unexpectedly, Conversation Relay keeps the call active and buffers up to 500 inbound events (or up to 10 seconds of call history) during the outage. Conversation Relay then opens a new WebSocket connection to your application URL and sends a setup message with connectionType set to "reconnect". Immediately after the setup message, Conversation Relay replays the buffered events in order.

Conversation Relay reconnects when the WebSocket closes with code 1001, 1006, 1011, 1012, or 1013, or when your application stops answering pings for 45 seconds. Any other close code ends the call. For each drop, Conversation Relay makes up to 3 connection attempts, waiting 500 ms and then 1 second between them, and allows 3 seconds for each WebSocket upgrade. It stops at once if your server returns a 4xx response other than 408 or 429, or if the TLS certificate is invalid. A call can reconnect at most 5 times. If reconnection fails, the call ends with a failed status.

Automatic reconnection covers network drops and server restarts, not closes that you initiate. If your application closes the WebSocket itself, the call ends, so send an end message to end a call on purpose.

To preserve conversation context, store your session state in a shared cache, such as Redis, keyed by sessionId. When you receive a setup message with connectionType set to "reconnect", load the existing state by using the sessionId instead of starting a new conversation.

If the agent's synthesized speech bleeds into the caller's microphone, the speech-to-text (STT) provider might transcribe the agent's own words. The agent can then interrupt itself or respond to its own speech.

To prevent this, turn on echo prompt filtering. Set the echoPromptFiltering attribute to true in your <ConversationRelay> TwiML, or update the setting during the conversation by sending a config message.

If you use echo prompt filtering, we recommend using ElevenLabs as your TTS provider. ElevenLabs provides word-level timing, which improves filtering accuracy compared to other providers.

Use the language message to switch languages dynamically during a session. When you switch languages during a session via a WebSocket message, Conversation Relay uses the settings, including the voice, that you configured for the new language in TwiML or in an earlier update. If the session doesn't have settings for that language, Conversation Relay rejects the switch with an error and the call stays on its current language.

• Initial Language Configuration: Declare each language that you plan to switch to in TwiML, with the voice that you want for it. This keeps a consistent voice experience across languages. For example, if you don't choose the Spanish voice yourself, switching from English (en-GB) with a male voice to Spanish (es-ES) may result in a female voice.
• Languages added during the call: To switch to a language that TwiML didn't declare, add it with its settings in full in the same message or an earlier one. Conversation Relay doesn't fill in default settings during the call. See Add a language and switch to it.
• Voice updates mid-session: To change the voice or other language settings after the session starts, send a config message. See Update configuration during a call.

Configuring a voice for each language in TwiML before the call begins ensures consistent voice behavior when you switch languages.

Send each text token incrementally with "last": false. When the LLM indicates that the response is complete (for example, when response.finish_reason() equals "stop"), mark that final token with "last": true. For messages with complex punctuation, consider breaking the response into smaller chunks.

Example:

1
{ "type": "text", "token": "Hello", "last": false }
2
{ "type": "text", "token": " world", "last": false }
3
{ "type": "text", "token": "!", "last": true }

When the entire response is generated as a single complete sentence, mark the token with "last": true.

Example:

{ "type": "text", "token": "Hello world!", "last": true }

You can record Conversation Relay calls using different recording methods outlined in the Recordings resource documentation. For examples on how to record calls, please review the Recording a Phone Call with Twilio help documentation.

Conversation Relay integrates with Conversation Intelligence (classic) for native virtual agent observability and performance monitoring. To enable this feature, you need to set the intelligenceService attribute in the <ConversationRelay> noun as documented below.

For more details on getting started with the Conversation Relay and Conversation Intelligence (classic) integration, please see this guide.

This guide covers a feature that can support the following use cases:

You can use the feature in this guide to optimize real-time speech responses for autonomous customer support agents. By adhering to text normalization rules and streaming LLM tokens instantly, you ensure your support bot provides fluid, human-like answers when helping users manage returns, track orders, or answer service FAQs. To learn more advanced features that you can use with customer support virtual agents, see AI agents.

You can utilize the strategies in this guide to test and implement high-performing AI/ML transcription tailored to your application's distinct acoustic environments. By comparing options like Google and Deepgram, you can discover which provider delivers the highest accuracy for both clean and noisy audio streams. You can configure these speech-to-text providers and models directly within your TwiML setup using the transcriptionProvider and speechModel attributes of the <ConversationRelay> noun. To learn more features that you can use with AI/ML transcription, see AI/ML transcription.

Explore the following guides to build on what you've learned in this guide:

• Picking a voice: Learn about the different Text-to-Speech voices you can use with Conversation Relay to create a unique brand experience.
• Update configuration during a call: Change language, voice, and session settings over your WebSocket connection without ending the call.

Find reference documentation for the topics on this page.

• TwiML™ Voice: <ConversationRelay>: Reference documentation for configuring speech-to-text engines, text-to-speech providers, and initial language keys.
• Conversation Relay WebSocket Messages: Reference documentation for handling structured JSON messages for text tokens, language switching, and connection states on your server.
• Recordings resource: Reference documentation for programmatically capturing, storing, and managing the audio logs of active voice calls.
• Conversation Intelligence (classic) Integration API: Reference documentation for connecting virtual agent streams directly to intelligence services for automated compliance monitoring and analytics.