From AI Agent to Human Agent: ElevenLabs Call Handoffs with Twilio

From AI Agent to Human Agent: ElevenLabs Call Handoffs with Twilio

Conversational AI agents are great at handling repeatable support questions, but even the best voice bot needs a graceful way to hand the caller to a human when the issue becomes complex, sensitive, or simply unresolved.

In this post, we'll walk through a reference integration that connects an ElevenLabs voice agent to Twilio Programmable Voice, then lets that agent hand the active phone call back to Twilio routing when escalation is needed. The reference destination is Twilio Flex, but the core pattern is broader: pass the original Twilio parent Call SID into ElevenLabs, let the ElevenLabs agent decide when to escalate, and update that original Twilio Call resource with the next TwiML instruction.

Voice AI architectures are becoming more modular. Some teams want to use Twilio's voice AI products directly. Others want to bring an external real-time agent provider into the call flow, while still relying on Twilio for telephony, routing, agent workflows, and customer context. This reference pattern is for that second case.

The full setup, code, agent prompts, tool templates, Studio sample, and tests are available in the reference repo: https://github.com/rbangueses/twilio-elevenlabs-call-handoff-blueprint

Why this pattern matters

The hard part of AI-to-human escalation is not only deciding when the AI should stop. It is also preserving control of the live call and carrying useful context into the next experience.

In this pattern, Twilio keeps ownership of the original inbound call. ElevenLabs handles the real-time AI conversation. When the ElevenLabs agent decides to escalate, it calls a webhook tool that asks Twilio to update the original parent call.

That gives you three useful capabilities:

• Interoperability: ElevenLabs can be the real-time agent provider while Twilio remains the voice, routing, Flex, and customer engagement layer.
• Containment: The caller is not trapped in the AI experience. The AI agent has a clear escape hatch back to Twilio routing.
• Context: The handoff can include intent, summary, direction, called number, customer number and other custom parameters so the next destination is not starting cold.

In this example we will use Twilio Flex as the human agent desktop for escalations. Flex is a natural reference destination because it gives human agents a voice workspace. But the architectural point is broader: the escalation target is whatever Twilio route you choose next.

Before you start

To follow this blueprint end to end, there are two prerequisites: some familiarity with the products involved, and the accounts and resources needed to actually run the integration.

Familiarity

• Twilio Programmable Voice, TwiML, and Twilio Functions (Serverless). You should be comfortable pointing a phone number at a Voice webhook and reading a TwiML response.
• Twilio TaskRouter and Flex, at least at the level of workflows, task attributes, and voice tasks, if you plan to test Pattern A or Pattern B.
• Twilio Studio, if you plan to test the Studio-owned journey in Pattern A.
• ElevenLabs Conversational AI agents, prompts, dynamic variables, and webhook tools. You should be comfortable creating an agent, editing its prompt, and configuring a tool from the ElevenLabs dashboard.
• Basic Node.js, since the Twilio Functions and tests are written in JavaScript.

Accounts and resources

• A Twilio account with a working voice-enabled phone number and permission to deploy Twilio Functions.
• Twilio Flex enabled on the account, plus a TaskRouter workspace and workflow, if you plan to route escalations into Flex (Pattern A or Pattern B).
• An ElevenLabs account with Conversational AI enabled and at least one agent you can edit.
• An ElevenLabs API key with permission to call the register-call endpoint.
• A custom TwiML-controlled destination or downstream voice workflow if you plan to adapt Pattern C for a non-Flex destination.
• Optional: Twilio Conversation Orchestrator, Conversation Memory, and Conversation Intelligence, if you want to layer in cross-channel context and analytics.

How the handoff works

At a high level, the call flow is:

1. A caller dials a Twilio phone number.

2. Twilio invokes a Function such as /voice.

3. The Function calls the ElevenLabs register-call API, passing the original Twilio CallSid and handoff metadata as ElevenLabs dynamic variables.

4. ElevenLabs returns TwiML that Twilio uses to bridge the caller into the ElevenLabs agent.

5. The ElevenLabs agent handles the self-service part of the conversation.

6. If escalation is needed, the agent calls the escalate_to_human webhook tool.

7. The webhook tool posts the parent Call SID, intent, and summary to a protected Twilio Function.

8. Twilio updates the original parent call with the next TwiML instruction.

The parent Call SID is the key. The Twilio Function that registers the call with ElevenLabs passes the original inbound CallSid into ElevenLabs as a dynamic variable:

ElevenLabs exposes that value to the agent, and it can be templated directly into tool parameters through the {{parent_call_sid}} placeholder. When the agent escalates, the webhook tool sends that value back to Twilio. Twilio then updates that exact Call resource.

For a Flex handoff, the update is an <Enqueue> instruction:

That moves the live caller into the TaskRouter workflow and creates a Flex voice task with the handoff context as task attributes.

Because Twilio never gives up ownership of the parent Call, the escalation is a call update rather than a transfer. That distinction matters: it is what lets Studio, TaskRouter, and Programmable Voice all remain valid handoff destinations behind the same tool.

What you can do with it

The reference implementation covers three routing patterns.

Pattern A: Studio owns the journey

Use this when Studio should remain the voice journey owner. Studio receives the inbound call, uses a TwiML Redirect widget to send the caller to the /studio_voice Function (which registers the call with ElevenLabs), then resumes the same Studio execution when the ElevenLabs agent escalates through /studio_escalate. From there, Studio can run Send to Flex or route the caller somewhere else.

This is a good fit when you already have IVR logic, reporting, routing branches, or post-AI decisions in Studio.

Pattern B: TaskRouter receives the handoff

Use this when you want the smallest direct path into Flex or another TaskRouter-powered destination. The Twilio number points directly at /voice, ElevenLabs handles the AI conversation, and /escalate updates the parent call with <Enqueue>.

This is the most compact version of the pattern and the easiest one to test end to end.

Pattern C: Twilio routes somewhere else

Use this when the escalation destination is not Flex or TaskRouter. The same escalate_to_human tool can still send Twilio the parent Call SID and handoff summary, but the Twilio Function can update the parent call with a different TwiML instruction such as <Dial>, <Conference>, <Sip>, or <Redirect>.

In the reference repo, Pattern C is documented as an extension pattern rather than a fully packaged direct-transfer feature. The important design choice is where the receiving system will get context. TaskRouter task attributes solve that for Flex-style workflows, but a custom voice destination may need a different context channel. In production, a common approach is to store the full handoff payload in an external datastore keyed by handoffId, parentCallSid, or another correlation ID, then pass only that identifier to the downstream app, CRM, custom desktop, or voice workflow.

Outbound Calls

The same pattern applies when Twilio places the call instead of receiving one, so the ElevenLabs agent can drive proactive use cases such as appointment reminders, delivery confirmations, payment collection, satisfaction surveys, or lead qualification, with the same escalation path back to a human when the conversation needs it.

The reference implementation exposes a dedicated /outbound Function for this case. The flow mirrors the inbound path:

1. Your application creates an outbound call via the Twilio REST API and points its Voice webhook at /outbound.

2. When Twilio dials the destination and the call is answered, Twilio invokes /outbound with the new CallSid, From, and To.

3. /outbound passes call_direction=outbound as well as the same dynamic variables used for inbound, so parent_call_sid still points at the Twilio parent leg.

4. ElevenLabs returns TwiML, Twilio bridges the answered call into the ElevenLabs agent, and the conversation begins.

5. If the agent needs a human, escalate_to_human fires exactly as it does for inbound, and Twilio updates the same outbound parent Call with <Enqueue>, a Studio return, or a custom TwiML destination.

Some things to consider for outbound calling:

• Because your application is the one starting the call, you already know who you are dialing and why. You can pass that context (customer name, account ID, reminder subject, order number) as additional ElevenLabs dynamic variables when /outbound calls register-call, so the agent opens the call with the right script rather than asking the customer to identify themselves.
• Outbound automated voice traffic is subject to consent, calling-hours, and answering-machine rules that inbound traffic is not. You should combine the blueprint with Twilio's answering-machine detection, caller ID configuration, and any consent capture your use case requires before dialing at volume.
• Because the parent Call SID mechanism is identical, an escalated outbound call can land in the same Flex queue, the same Studio Flow, or the same custom TwiML-controlled route as an escalated inbound call. Human agents do not need a separate workflow for AI-initiated calls.

WhatsApp Business Calling

The same inbound handoff patterns can also apply to Twilio WhatsApp Business Calling. In that case, the WhatsApp sender's voice entry point is a TwiML Application rather than a phone-number webhook. The TwiML App can point directly to /voice for the TaskRouter/Flex path, or to the published Studio Flow webhook for the Studio-owned path.

From there, the same ElevenLabs register-call and escalate_to_human mechanics apply.

Adding customer context with Conversation Memory

The baseline handoff makes sure the call can move from AI self-service to human support. Conversation Memory adds another layer: the caller should not have to start from scratch every time they contact you.

With Memory enabled, Twilio can maintain persistent customer context across calls and channels. In the current blueprint, /voice, /studio_voice, and /outbound already pass customer_number to ElevenLabs as a dynamic variable. The ElevenLabs agent can then expose a recall_customer_memory webhook tool that points at /memory_recall. That Function resolves the Memory profile on demand by phone number and returns relevant prior context to the agent.

The agent should call that tool only when prior context would help the current conversation, for example when the caller asks what happened previously or when a recent support issue might help the agent avoid asking the customer to repeat themselves.

This becomes more powerful when other channels are involved. If the customer previously contacted you over SMS, WhatsApp, RCS, chat, voice, or another captured Twilio channel, Conversation Orchestrator can group those communications into conversations and link them to a Memory profile. An ElevenLabs voice agent can then recall relevant cross-channel context during a live voice call.

Memory can also improve escalation. The escalate_to_human payload can pass customer and call identifiers along with the AI summary so Flex, TaskRouter, Studio, or another downstream system can retrieve or display the right context.

Where Orchestrator and Conversation Intelligence fit

Conversation Memory and Conversation Intelligence are independent capabilities.

Memory stores and recalls customer context. Conversation Intelligence analyzes conversations for real-time or post-conversation signals such as summaries, sentiment, next-best-response, QA, or custom operator outputs.

In this pattern, both can use Conversation Orchestrator as the capture and configuration layer. Orchestrator turns voice and messaging traffic into normalized conversations, links those conversations to a Memory Store, and can also attach Intelligence configurations.

That means enabling the optional Memory path can also create the foundation for Conversation Intelligence. Once the same Orchestrator configuration is capturing the relevant voice or messaging traffic, you can attach an Intelligence configuration to run real-time or post-call analysis without changing the ElevenLabs handoff mechanics.

You might start with:

- Memory recall during the AI call.

- A handoff summary passed into Flex task attributes.

- Post-call Conversation Intelligence summaries.

- Real-time Intelligence operators for escalation risk, sentiment, or compliance signals.

Each piece can be adopted separately. The handoff pattern does not require all of them, but it gives you a clean place to connect them.

Try the reference implementation

The repo includes the building blocks needed to test the pattern:

- Twilio Functions for /voice, /studio_voice, /outbound, /escalate, /studio_escalate, /start_outbound, /start_studio_outbound, /outbound_status, and /memory_recall.

- An ElevenLabs agent prompt with clear escalation triggers.

- An escalate_to_human webhook tool template ready to import into the ElevenLabs dashboard.

- A Studio Flow sample for the Studio-owned journey.

- Setup, architecture, and testing docs that mirror the routing patterns.

Use the repo README as the implementation guide: https://github.com/rbangueses/twilio-elevenlabs-call-handoff-blueprint

The most useful first test is Pattern B: point a Twilio number to /voice, call in, talk to the ElevenLabs agent, ask for a human, and confirm that a Flex voice task appears with the handoff summary in task attributes.

After that, test Pattern A if Studio should own the journey, Pattern C for a custom TwiML-controlled destination, or add Memory if the agent needs prior customer context.

Wrap-up

This integration is about interoperability and context.

ElevenLabs can provide the real-time AI agent experience. Twilio can keep ownership of the live phone call, routing, TaskRouter workflow, Flex handoff, and optional customer context services. The parent Call SID is the bridge that makes the handoff practical: it lets the AI agent request escalation while Twilio updates the original caller leg.

That separation keeps the architecture flexible. You can start with a direct Flex handoff, move routing ownership into Studio, route to another TwiML destination, or add Conversation Memory and Conversation Intelligence when the use case calls for richer context.

AI containment is not only about knowing when the bot should stop. It is about making sure the next step is live, routed, and context-aware.

Ricardo Bangueses is a Product Specialist at Twilio with a strong background in the Customer Engagement and Contact Center industry. He can be reached at rbangueses [at] twilio.com