How to build multilingual AI agents for customer support

How to build multilingual AI agents for customer support

Language isn’t always linear. A customer might initiate a call in English but switch to Spanish midway through the conversation. It happens, but most voice agents keep right on answering in English. The caller repeats themselves or perhaps returns to a non-native English, or they ask for a human—either way, your AI agent has let them down.

It doesn’t need to be this difficult, though. Nor do you need to hire dozens of bilingual agents or translate your entire help center. AI agents make multilingual customer support scalable for businesses of all sizes and budgets. A single agent can serve customers in many languages, but it needs to be able to detect when a caller switches languages and follow along when it does. 

Here's how to build an AI agent that detects a customer's language, switches mid-conversation, and remembers the preference for next time (on the phone and in messaging).

What is a multilingual AI agent?

A multilingual AI agent is an AI agent that detects and responds to customers in more than one language. Smart multilingual agents recognize which language a customer is using and switch its own language to match.

This is simpler in messaging and chat functions. Modern LLMs read and write dozens of languages, so a text-based agent can usually respond in whatever language the customer writes in. Easy.

But voice is harder. A phone conversation runs through three systems: 

1. Speech-to-text turns the caller's voice into words

2. The LLM reads and decides what to say

3. Text-to-speech turns the reply back into audio

The LLM might handle Spanish perfectly, but if speech-to-text is still listening for English, the transcript it hands the LLM won’t work. All three layers have to switch together.

How to build a multilingual AI agent

We’ll be using Twilio ConversationRelay, which handles speech-to-text and text-to-speech for voice AI agents. It also lets you connect the LLM of your choice. First, you’ll configure languages before you detect them, and then you’ll need to detect them before you switch.

1. Choose speech providers that cover your languages

Every language your agent might hear needs a speech-to-text provider that can transcribe it and a voice that can speak it. Language availability, coverage, and quality vary depending on your provider, so start with the languages your customers speak, then double-check each one is supported.

In ConversationRelay, you add one <Language> element for each language a call might use. Each element maps a language code to its own text-to-speech provider, voice, transcription provider, and speech model. That means your Spanish voice can come from a different provider than your English one if that's what sounds best for you.

Adding a <Language> element doesn't activate it, though. It tells ConversationRelay what to use when the conversation switches to that language.

2. Detect the caller's language automatically

ConversationRelay detects a caller's language automatically when you set the speech-to-text language to multi (this requires Deepgram as the transcription provider). The speech-to-text provider identifies the language of each utterance, and every transcribed message your agent receives arrives tagged with that language in a lang field.

Again, that’s per utterance. Your agent doesn't have to guess the language from the words, and it catches the moment a caller switches partway through a call, because the tag changes on the next utterance.

• Tags arrive as primary language codes: Detected languages come back as en or es, without a regional suffix like en-US. If your code maps languages to specific voices, account for that.

• Text-to-speech can detect too: Setting the text-to-speech language to multi lets ElevenLabs detect the language from the text your agent sends, so the voice follows the reply.

If you know which languages to expect, ConversationRelay also supports language hints. With Deepgram's Flux speech model, your <Language> elements act as hints that bias recognition toward the languages you've configured. Flux also pairs with a newer attribute, ignoreBackchannel, which filters filler words like “uh-huh” and “okay” so they don’t interrupt the agent mid-reply. Twilio says it covers English, Spanish, French, German, and more.

3. Switch languages mid-conversation

ConversationRelay switches languages mid-conversation through a language message your agent sends over the WebSocket connection. This changes the speech-to-text language or the text-to-speech language (or both) without restarting the call.

A typical flow looks like this:

1. The caller switches to Spanish. The next transcribed message arrives tagged es.

2. Your agent notices the change. Your code compares the new tag to the current language.

3. Your agent sends a language message. It sets the new transcription and text-to-speech languages.

4. The conversation continues in Spanish. The voice changes to the one you configured for Spanish, and the agent keeps going.

For finer control, you can also set the language on individual text tokens your agent sends. That helps when a reply mixes languages, like an English-speaking agent reading out a Spanish street address.

4. Keep the LLM consistent across languages

Switching the voice is only half the job. The LLM also needs clear instructions about what to do when the language changes.

These prompt-design rules can help:

• Respond in the customer's current language: Pass the detected language into the agent's context on every turn, and instruct it to answer in that language.

• Keep facts language-independent: Order numbers, prices, and policy details should come from your tools and knowledge base unchanged, whatever language the conversation is in.

• Decide how to handle mixed input: Some customers switch languages mid-sentence. Tell the agent whether to follow the most recent language or the dominant one.

Watch our October 2026 webinar, "One agent, many languages", to learn how to implement these prompt-design and fallback decisions on ConversationRelay.

5. Remember the customer's language for next time

When your customer spoke Spanish last week, it’s safe to assume they’ll prefer the same this week. Store the preference.

Twilio Conversation Memory keeps traits and observations on each customer's profile. Twilio's Agent Connect and Flex handoff tutorial shows once you’ve updated the language trait on the customer's profile to Spanish, the agent communicates with that customer in Spanish.

Simple as that. On the next call, your agent reads the trait and starts in Spanish from the first greeting.

Multilingual agents across voice and messaging

Customers don’t follow a linear customer journey. They bounce between channels. But their language preferences shouldn’t be limited to a single channel. It’s safe to assume (by default at least), that when your customer prefers to talk over the phone in Spanish, they’ll probably want to text in Spanish, too. 

Twilio Agent Connect routes Voice, SMS, WhatsApp, RCS, and Chat through a single message handler, so one agent with one set of language rules covers every channel. On messaging channels, the LLM reads the customer's language directly from the text. On voice, ConversationRelay handles detection and switching as described above.

The language trait on the customer's profile ties it together. A customer who texts in Portuguese on Monday and calls on Thursday reaches an agent that already knows to speak Portuguese.

Escalation: Handing off to a human who speaks the language

The escalation path should build off what your agents have already learned. That means when a Spanish-speaking customer wants to talk to a person, you route them to a Spanish-speaking representative. 

Agent Connect's built-in handoff tool accepts optional routing attributes, and those attributes surface as TaskRouter task attributes in Twilio Flex. Pass the detected language as an attribute, and Flex can route the conversation to an agent who speaks it.

The customer stays on the same call or chat throughout, and the conversation history travels with the handoff.

Meet customers in their language (on their channel)

Work smarter so your customers work less harder. A multilingual agent lets you meet customer in their language on their preferred channel. Configure your languages, let detection run on every utterance, switch the instant the customer does, and remember the preference for next time. 

If you’re worried about Voice AI speed, check out our guide to core latency in AI voice agents.

Frequently asked questions

What are the best platforms for building multilingual AI agents that can detect language and switch mid-conversation?

Twilio ConversationRelay detects a caller's language on every utterance when set to multi-language mode with Deepgram, and switches speech recognition and voice languages mid-call through a language message. Agent Connect extends the same agent to SMS, WhatsApp, RCS, and Chat.

How does an AI voice agent detect which language a caller is speaking?

Twilio ConversationRelay uses the speech-to-text provider to identify the language of each utterance and tags every transcribed message with it. Your agent reads the tag and can switch languages as soon as the caller does.

Can an AI agent switch languages in the middle of a phone call?

Twilio ConversationRelay lets your agent send a language message that changes the speech-to-text and text-to-speech languages mid-call without restarting the conversation. The voice switches to the one configured for the new language.

How many languages can a multilingual AI agent support?

Twilio ConversationRelay supports the languages offered by its speech-to-text and text-to-speech providers, configured with one language element per language. Coverage and quality vary by provider, so confirm each language your customers speak.

How do you route a multilingual customer to a human agent who speaks their language?

Twilio Agent Connect's handoff tool passes routing attributes into Flex as TaskRouter task attributes. Include the detected language, and Flex can route the conversation to an agent who speaks it, with the history attached.