Best Practices for Conversation Relay Provider Failover

Time to read:

November 04, 2026

Written by

Daniel Jansson

Twilion

Reviewed by

Hemkiran Gadey

Twilion

Brandon Hawkins

Twilion

Lenore Files

Twilion

Paul Kamp

Twilion

Best Practices for Conversation Relay Provider Failover

When you're scaling Voice AI agents in production using Twilio Conversation Relay, keeping your architecture uninterrupted is critical. That’s especially true during high-volume events like Black Friday or Cyber Monday, or sudden traffic spikes. High load or unexpected provider degradation can introduce latencies, connection drops, or request failures. To harden your Conversation Relay infrastructure against these risks, you need an application-level failover strategy.

Conversation Relay uses two speech providers on every call: a transcription (ASR/STT) provider that turns the caller's speech into text, and a text-to-speech (TTS) provider that speaks your agent's replies. Each one is a third-party service, such as Deepgram, Google, ElevenLabs, or Amazon. Sometimes, a service might be slow, unavailable, or reject a request.

This guide covers:

How a provider failure reaches your application, at startup and mid-call.
Which error codes to look for.
How to return new <ConversationRelay> TwiML from your action URL that fails over to a different provider, so the caller stays on the line.
How to deal with failures in the middle of a call by sending an end message with handoffData, then switching providers in the TwiML that follows.

How failures reach you

Where a failure shows up depends on which provider failed and when it failed:

When	Provider	What Conversation Relay does	Where you see it
Startup	Transcription (ASR)	Ends the session.	`action` URL: `SessionStatus=failed` and an `ErrorCode`
Startup (welcome greeting)	Text-to-speech (TTS)	Usually ends the session. If the failure arrives just after the call connects, it's reported like a mid-call TTS failure instead (next row).	`action` URL: `SessionStatus=failed` and an `ErrorCode`, or a WebSocket `error` message
Mid-call	Transcription (ASR)	Ends the session.	`action` URL: an `ErrorCode` (`SessionStatus` may be `failed` or `ended`)
Mid-call	Text-to-speech (TTS)	Keeps the call up. The failed reply is not spoken, and later replies are tried again.	WebSocket `error` message only. Your server decides whether to end the session.

Two rules follow from this table:

In your action URL handler, check ErrorCode, not only SessionStatus. Some mid-call failures report SessionStatus=ended and still include an ErrorCode.
Handle TTS failures in your WebSocket server. When the agent's voice stops working mid-call, the call stays connected and the caller hears silence. Conversation Relay won't end the call for you. Your server has to notice the error messages and send end (see the Mid-call failures section, below).

Prerequisites for failover

Before implementing custom failover logic, your Conversation Relay application should implement a few basic best practices. These fundamentals will help you build a robust production service.

Always set an `action` URL on `<Connect>`

Failover only works when an action URL is set on <Connect>. When the Conversation Relay session ends, for any reason, Twilio sends a request to that URL, and the call carries on with the TwiML you return. Without an action URL, the call moves on to whatever verb follows <Connect> in the original document, and you can't vary that by error.

<Response>
  <Connect action="https://example.com/relay-ended?attempt=1">
    <ConversationRelay url="wss://example.com/agent" ... />
  </Connect>
</Response>

Name your providers explicitly

Always set ttsProvider and transcriptionProvider, even when you're using the defaults. That way every TwiML document says exactly which provider it uses, and your failover TwiML can switch to a different one.

If you leave out the TTS or transcription settings for a language, Conversation Relay fills them from a default (ElevenLabs with a default voice for TTS; Google or Deepgram for transcription). That default is not something you can fail over from or to on purpose. Set ttsProvider next to voice as well, rather than letting the provider be inferred from the voice name.

Track the attempt number

Put the attempt number, and which providers you've already tried, in the action URL query string (for example ?attempt=2&tts=google). Your handler then knows where it is in the failover chain and can't end up in a loop (see the Avoiding failover loops section for more).

Watch your `action` URL parameters

When the session ends, Twilio sends the standard Voice call parameters (CallSid, AccountSid, From, To, CallStatus, and so on) plus these Conversation Relay parameters:

Parameter	Present	Description
`SessionId`	Always	ID of the Conversation Relay session that ended.
`SessionStatus`	Always	`completed`: the caller hung up. `ended`: the session was ended, normally by your `end` message. `failed`: the session ended because of an error.
`SessionDuration`	Always	How long the session lasted, in seconds.
`ErrorCode`	On error	Twilio error code. Branch on this.
`ErrorMessage`	On error	Human-readable description. It can include the provider's own error text. Log it, but don't branch on it, because the wording can change.
`HandoffData`	When you sent it	The `handoffData` string from your `end` message, passed through unchanged.

Error codes to watch

Some error codes you’ll get back from the webhook suggest it’s time for you to fail over to another provider. This section has an overview of the types of codes you may see, plus the individual error codes.

Provider failures: candidates for failover

Code	Meaning	Typical cause	Recommended action
64114	Unable to connect to ASR provider	The transcription provider is unreachable, timed out during setup, or failed during the call.	Fail over transcription to a different `transcriptionProvider`.
64111	TTS provider service error	The TTS provider returned a server error (5xx), timed out, or dropped the connection. This usually means a provider outage or slowdown.	Fail over TTS to a different `ttsProvider`.
64112	TTS provider conversion error	The TTS provider rejected the request, for example because the voice ID doesn't exist, the credentials or quota are invalid, or the text or settings aren't supported. `ErrorMessage` (or the WebSocket `description`) includes the provider's reason.	Fail over TTS to keep the caller on the line, and check your voice and provider configuration. This error usually happens again with the same settings. Exception: if the message contains `input_timeout_exceeded`, see the 64112 section below.
64103	Internal server error	Something failed inside Twilio, not at your speech provider. Provider failures are reported with 64111, 64112, and 64114 instead.	Retry once with the same providers. If the error keeps happening, contact Twilio Support with the `CallSid`.

Upgrading an existing handler? Earlier versions of Conversation Relay reported many provider failures as 64103, and Google or Amazon voice rejections as 64106. Handlers that retry 64103 once keep working, but branching on 64111, 64112, and 64114 is now enough to catch provider failures.

Other codes you may see in the same handler

Code	Meaning	Suggested handling
64102	Unable to connect to WebSocket URL	Fail over to a backup WebSocket URL (`url=`), not to a different speech provider.
64105	WebSocket session ended	Your server closed the WebSocket without sending `end`. Send `end` instead (see the Mid-call failures section).
64107	Invalid message received	Your server sent a malformed message. Fix the server.
64104	Max call duration reached	Expected behavior. No failover.

Startup failures

A startup failure means the session ended before the conversation got going. Usually the caller has heard nothing yet, or only part of the welcome greeting. Conversation Relay calls your action URL with SessionStatus=failed and an ErrorCode, and the caller stays on the line while you return new TwiML.

Original TwiML (primary providers)

<Response>
  <Connect action="https://example.com/relay-ended?attempt=1&amp;stt=deepgram&amp;tts=elevenlabs">
    <ConversationRelay
        url="wss://example.com/agent"
        transcriptionProvider="Deepgram"
        speechModel="nova-3-general"
        ttsProvider="ElevenLabs"
        voice="YOUR_ELEVENLABS_VOICE_ID"
        language="en-US"
        welcomeGreeting="Hi! Thanks for calling. How can I help you today?" />
  </Connect>
</Response>

Escape & as & inside XML attribute values.

Transcription (ASR) failed at startup: `ErrorCode=64114`

Keep the TTS provider and switch the transcription provider:

<Response>
  <Connect action="https://example.com/relay-ended?attempt=2&amp;stt=google&amp;tts=elevenlabs">
    <ConversationRelay
        url="wss://example.com/agent"
        transcriptionProvider="Google"
        speechModel="telephony"
        ttsProvider="ElevenLabs"
        voice="YOUR_ELEVENLABS_VOICE_ID"
        language="en-US"
        welcomeGreeting="Hi! Thanks for calling. How can I help you today?">
      <Parameter name="failoverReason" value="stt-64114" />
    </ConversationRelay>
  </Connect>
</Response>

When you switch transcription providers, also change speechModel to a model the new provider supports. A Deepgram model name sent to Google, or the other way round, fails with 64106.

TTS failed at startup: `ErrorCode=64111` or `64112`

Keep the transcription provider and switch the TTS provider and voice:

<Response>
  <Connect action="https://example.com/relay-ended?attempt=2&amp;stt=deepgram&amp;tts=google">
    <ConversationRelay
        url="wss://example.com/agent"
        transcriptionProvider="Deepgram"
        speechModel="nova-3-general"
        ttsProvider="Google"
        voice="en-US-Journey-O"
        language="en-US"
        welcomeGreeting="Hi! Thanks for calling. How can I help you today?">
      <Parameter name="failoverReason" value="tts-64111" />
    </ConversationRelay>
  </Connect>
</Response>

Voices belong to a provider. When you change ttsProvider, choose a voice from the new provider's catalog. Pick your backup voice in advance, so the agent sounds as close as possible to the primary.

The backup provider must also support your language. Providers don't all serve the same languages, and a backup that doesn't (for example Amazon with zh-TW or he-IL) is rejected with 64101, which means your failover TwiML fails too. Test every backup voice for each language you support.

The <Parameter> elements are passed to your WebSocket server in the setup message under customParameters. That way your agent knows it's running on a failover session, and can, for example, keep the response style shorter or log the event.

Example `action` URL handler

Here's a minimal Node.js (Express) handler. It picks the next provider, limits the number of attempts, and falls back to a human agent when nothing else works:

const express = require('express');
const app = express();
app.use(express.urlencoded({ extended: false }));

const STT = {
  deepgram: { provider: 'Deepgram', speechModel: 'nova-3-general' },
  google:   { provider: 'Google',   speechModel: 'telephony' },
};
const TTS = {
  elevenlabs: { provider: 'ElevenLabs', voice: 'YOUR_ELEVENLABS_VOICE_ID' },
  google:     { provider: 'Google',     voice: 'en-US-Journey-O' },
  amazon:     { provider: 'Amazon',     voice: 'Joanna-Neural' },
};
const NEXT_STT = { deepgram: 'google', google: 'deepgram' };
const NEXT_TTS = { elevenlabs: 'google', google: 'amazon', amazon: 'elevenlabs' };
const MAX_ATTEMPTS = 3;

const ASR_CODES = new Set(['64114']);
const TTS_CODES = new Set(['64111', '64112']);
const RETRY_CODES = new Set(['64103']);
const FAILOVER_REASONS = new Set(['stt-failure', 'tts-failure', 'internal-error']);

app.post('/relay-ended', (req, res) => {
  const { SessionStatus, ErrorCode, ErrorMessage, HandoffData, CallSid } = req.body;
  const attempt = Number(req.query.attempt || 1);
  let stt = req.query.stt || 'deepgram';
  let tts = req.query.tts || 'elevenlabs';
  res.type('text/xml');

  // The caller hung up: nothing to do.
  if (SessionStatus === 'completed') return res.send('<Response/>');

  // A mid-call handoff from your WebSocket server (Mid-call failures section). Parse HandoffData first.
  let handoff = {};
  try { handoff = HandoffData ? JSON.parse(HandoffData) : {}; } catch (_) { /* not JSON */ }

  // Provider failures from ErrorCode, or from your own handoff reasonCode (Mid-call failures section).
  const reason = handoff.reasonCode
    || (ASR_CODES.has(ErrorCode) ? 'stt-failure'
      : TTS_CODES.has(ErrorCode) ? 'tts-failure'
      : RETRY_CODES.has(ErrorCode) ? 'internal-error'
      : null);

  if (ErrorCode) console.warn(`CallSid=${CallSid} ErrorCode=${ErrorCode} ErrorMessage=${ErrorMessage}`);

  // Only the explicit failover reasons start a new ConversationRelay. Everything else
  // (normal end, configuration errors like 64101/64106, live-agent, llm-failure, ...) is
  // not a provider failure, so it goes to handleNormalEnd.
  if (!FAILOVER_REASONS.has(reason)) return res.send(handleNormalEnd({ reason, ErrorCode, handoff }));
  if (attempt >= MAX_ATTEMPTS) return res.send(finalFallback());

  if (reason === 'stt-failure') stt = NEXT_STT[stt];
  if (reason === 'tts-failure') tts = NEXT_TTS[tts];
  // 'internal-error': retry the same providers once.

  res.send(relayTwiml({ attempt: attempt + 1, stt, tts, reason, handoff }));
});

function relayTwiml({ attempt, stt, tts, reason, handoff }) {
  const s = STT[stt], t = TTS[tts];
  const action = `https://example.com/relay-ended?attempt=${attempt}&amp;stt=${stt}&amp;tts=${tts}`;
  // A mid-call failover resumes the conversation, so don't greet the caller again.
  const greeting = handoff.conversationId
    ? 'Sorry about that, I had a brief technical issue. Where were we?'
    : 'Hi! Thanks for calling. How can I help you today?';
  return `<Response>
  <Connect action="${action}">
    <ConversationRelay url="wss://example.com/agent"
        transcriptionProvider="${s.provider}" speechModel="${s.speechModel}"
        ttsProvider="${t.provider}" voice="${t.voice}"
        language="en-US" welcomeGreeting="${greeting}">
      <Parameter name="failoverReason" value="${reason}" />
      <Parameter name="conversationId" value="${handoff.conversationId || ''}" />
    </ConversationRelay>
  </Connect>
</Response>`;
}

function finalFallback() {
  return `<Response>
  <Say>We're sorry, we're having trouble right now. Connecting you to an agent.</Say>
  <Dial>+15555550100</Dial>
</Response>`;
}

function handleNormalEnd({ reason, ErrorCode, handoff }) {
  // Not a provider failure: the default path. Branch on your own handoff reasons (Ending the call for other mid-call reasons section).
  if (reason === 'live-agent') return '<Response><Dial>+15555550100</Dial></Response>';
  if (reason === 'llm-failure') return '<Response><Say>Sorry, something went wrong. Goodbye.</Say><Hangup/></Response>';
  // A configuration error (64101, 64106, ...) won't be fixed by retrying: alert and end the call safely.
  if (ErrorCode) return finalFallback();
  return '<Response><Hangup/></Response>';
}

In production, escape every value you put into XML, and use the Twilio helper library or a TwiML builder rather than string templates where you can. Validate the X-Twilio-Signature header on every request to your action URL.

Mid-call failures

Sometimes, you’ll need to react to a failure in the middle of a call. The following sections detail failure modes you could run into during a live call and detail whether you should take action.

Mid-call transcription failure: no additional action needed in your WebSocket server

If the transcription provider fails during the call, Conversation Relay ends the session and calls your action URL with ErrorCode=64114 (or 64103 if the failure was inside Twilio). Depending on where in the call the failure happened, SessionStatus may be failed or ended. Your WebSocket connection is closed.

The action URL handler from the Example action URL handler section above already deals with this: it sees the ASR error code and returns TwiML with the other transcriptionProvider. To carry on the conversation instead of starting over, store your conversation state server-side (keyed by CallSid) and look it up again when the new session's setup message arrives. setup always includes callSid.

Mid-call TTS failure: your WebSocket server must act

A mid-call TTS failure does not end the session. Conversation Relay sends an error message to your WebSocket server, doesn't speak that reply, and keeps the call connected. If the provider is down, every later reply fails the same way, and the caller hears silence until they hang up.

The error message comes in one of two forms, depending on the provider:

{
  "type": "error",
  "description": "Error converting tokens to speech, code: 64111, error: <provider error>, tokens: Sure, your appointment is on Tuesday at 3 PM."
}

{
  "type": "error",
  "description": "Error playing media, code: 64112, error: TTS provider rejected the request due to invalid parameters: <provider error>"
}

The first form is used by most providers, including ElevenLabs and Deepgram. The second is used by Google and Amazon voices, and it doesn't include the unspoken text. Both include the error code (code: ...). When the first form also includes the text that couldn't be spoken (tokens: ...), you can pass it to the new session so the caller hears the reply they missed.

Recommended handling:

Read the code from description. Treat 64111 and 64112 as TTS failures, whichever form the message takes. Skip 64112 with input_timeout_exceeded (see the Error codes to watch section). Other codes, such as 64106 for a media URL that couldn't be played, aren't provider failures.
Decide when to fail over. A single 64111 can be a one-off. Two failures in a row, or any failure during the first turn (the welcome greeting), means the provider can't be relied on, so fail over.
Send an end message with handoffData that describes why you're ending and what the next session needs. handoffData must be a string, so call JSON.stringify on your object.
Conversation Relay ends the session and calls your action URL with SessionStatus=ended and HandoffData set to your string. Your handler returns TwiML that uses a different ttsProvider.

The end message:

{
  "type": "end",
  "handoffData": "{\"reasonCode\":\"tts-failure\",\"errorCode\":\"64111\",\"failedTtsProvider\":\"elevenlabs\",\"conversationId\":\"conv_8f2c\",\"unspokenText\":\"Sure, your appointment is on Tuesday at 3 PM.\"}"
}

Example WebSocket server logic (Node.js, `ws`)

const { WebSocketServer } = require('ws');
const wss = new WebSocketServer({ port: 8080 });

const TTS_ERROR_CODES = new Set([64111, 64112]);
const MAX_CONSECUTIVE_TTS_ERRORS = 2;

wss.on('connection', (ws) => {
  const session = { conversationId: null, consecutiveTtsErrors: 0, turns: 0, ending: false };

  ws.on('message', (raw) => {
    const msg = JSON.parse(raw);

    switch (msg.type) {
      case 'setup':
        // Resume an existing conversation after a failover, or start a new one.
        session.conversationId = msg.customParameters?.conversationId || newConversationId(msg.callSid);
        break;

      case 'prompt':
        session.turns++;
        session.consecutiveTtsErrors = 0; // the caller is being heard, so the conversation is still going
        handlePrompt(ws, session, msg);   // your LLM logic; replies are sent as {type:'text', token, last}
        break;

      case 'error':
        handleError(ws, session, msg.description || '');
        break;
    }
  });
});

function handleError(ws, session, description) {
  const match = description.match(/code:\s*(\d+)/);
  const code = match ? Number(match[1]) : null;
  // Matches both "Error converting tokens to speech" and "Error playing media" (Google/Amazon).
  const isTtsError = TTS_ERROR_CODES.has(code) && !description.includes('input_timeout_exceeded');
  if (!isTtsError || session.ending) return;

  session.consecutiveTtsErrors++;
  const failedOnGreeting = session.turns === 0;
  if (!failedOnGreeting && session.consecutiveTtsErrors < MAX_CONSECUTIVE_TTS_ERRORS) return;

  const unspoken = description.split(', tokens: ')[1] || ''; // empty for Google/Amazon messages
  session.ending = true;
  ws.send(JSON.stringify({
    type: 'end',
    handoffData: JSON.stringify({
      reasonCode: 'tts-failure',
      errorCode: String(code),
      conversationId: session.conversationId,
      unspokenText: unspoken.slice(0, 500),
    }),
  }));
}

Handling the handoff in your `action` URL

The handler above already reads HandoffData. When reasonCode is tts-failure, it moves on to the next TTS provider. It also passes conversationId through a <Parameter>, so the new session's setup message resumes the same conversation. Here's the TwiML for a mid-call TTS handoff that also replays the unspoken text:

<Response>
  <Connect action="https://example.com/relay-ended?attempt=2&amp;stt=deepgram&amp;tts=google">
    <ConversationRelay
        url="wss://example.com/agent"
        transcriptionProvider="Deepgram"
        speechModel="nova-3-general"
        ttsProvider="Google"
        voice="en-US-Journey-O"
        language="en-US"
        welcomeGreeting="Sorry about that, I had a brief technical issue. Sure, your appointment is on Tuesday at 3 PM.">
      <Parameter name="failoverReason" value="tts-failure" />
      <Parameter name="conversationId" value="conv_8f2c" />
    </ConversationRelay>
  </Connect>
</Response>

Putting unspokenText in welcomeGreeting means the caller hears the reply that failed. If you'd rather not, use a short "where were we?" greeting, or leave welcomeGreeting out and have your agent speak first once setup arrives.

Ending the call for other mid-call reasons

You can use the same end plus handoffData pattern for any mid-call problem your server detects, for example when your LLM provider is down, a tool call keeps failing, or the caller asks for a person. Set a different reasonCode (llm-failure, live-agent, ...) and branch on it in your action URL handler. Always prefer end to closing the WebSocket. If you close the WebSocket, the session fails with 64105 and you lose the HandoffData.

Avoiding failover loops

When you add failovers to your Conversation Relay app, you need to be careful that you don’t accidentally create a loop which cycles through fallback providers and never exits. This list contains our best practices for avoiding a failover loop.

Limit the number of attempts. Carry attempt in the action URL and stop after 2 or 3.
Change something on every attempt. Only retry identical TwiML for 64103, and only once.
Don't fail over on configuration errors (64101, 64106). They fail the same way on every attempt.
Always have a final fallback that doesn't use Conversation Relay, such as <Say> with a <Dial> to a human agent, <Enqueue>, <Record> for voicemail, or a polite <Hangup>. <Say> uses Twilio's own TTS, so it doesn't depend on the provider that just failed.
Return <Response/> when SessionStatus=completed, because the caller has already hung up.

Checklist

Looking for a cheatsheet on how to handle failovers in your production Conversation Relay app? This list contains a succinct version of our advice from this post, so you can check that your app is handling possible failure cases.

☐ Every <Connect> has an action URL that carries attempt and the current providers.
☐ ttsProvider, voice, transcriptionProvider, and speechModel are set explicitly.
☐ A backup voice and a backup speech model are chosen and tested for each language you support. A backup that doesn't serve the language fails with 64101.
☐ The action URL handler branches on ErrorCode (not only SessionStatus) and parses HandoffData.
☐ 64114 fails over transcription. 64111 and 64112 fail over TTS. 64103 retries once. 64101 and 64106 go to the fallback and get alerted on.
☐ Your WebSocket server watches for error messages with 64111 or 64112 (in both message forms) and sends end with handoffData.
☐ Conversation state is stored server-side and resumed through <Parameter name="conversationId">.
☐ The final fallback doesn't depend on Conversation Relay.
☐ ErrorCode, ErrorMessage, and CallSid are logged for every failover, so you can follow up with Twilio Support.

Conclusion

By implementing these failover strategies, your Conversation Relay Voice AI agents will be more resilient against speech provider outages, traffic surges, and transient degradations. Building a multi-provider fallback strategy helps ensure continuous, high-availability conversation flow – whether you’re dealing with high-volume events like Black Friday, or traffic on a random weeknight.

With these best practices in place, your voice infrastructure can be hardened for enterprise scale. We can’t wait to see – and hear – what you build!

Suggested resources

Daniel Jansson is a Tech Lead at Twilio, where they have worked since 2017 and helped create Conversation Relay. They spend a lot of their time making voice AI calls keep working when the speech-to-text or text-to-speech provider on the other end doesn't. They can be reached at djansson [at] twilio.com.

Related Resources

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.

Best Practices for Conversation Relay Provider Failover

Related Posts

Related Resources