Twilio Expands Conversations Layer for AI Agent Supervision

November 17, 2026
Written by
Ruma Nair
Twilion
Reviewed by
Jeff Eiden
Twilion
Paul Kamp
Twilion

AI agents now handle a growing share of customer interactions. They answer questions, complete tasks, and act on behalf of the companies that deploy them. That creates a new operational challenge. How do you know whether your AI agents are performing well when they engage with your customers?

Many AI observability tools look inside the agent, at traces, tool calls and latency. They take work to set up and maintain. You instrument each agent, build your own evals, and end up with a lot of data. This makes it hard to tell what the customer actually experienced, or whether the agent solved their problem with the right answer.

Twilio starts from the conversation instead. If your conversations already run on Twilio, Twilio has the customer-facing record of what happened. Conversation Intelligence operators review that record whether your agent runs on one model, several models, or an external agent platform. You don’t need to change your agent code or piece together data from different systems.

Today, we’re adding AI agent supervision to Twilio Conversations. It includes two Twilio-authored Conversation Intelligence operators. Task Completion shows whether the agent resolved the customer’s request. Hallucination Detection flags when the agent said something that conflicts with your knowledge base or with the conversation itself. Both run when a conversation ends. Conversation Insights then shows patterns across all your conversations, so you can see what to fix in your agents.

Check outcome and accuracy across your conversations

Supervising AI agents starts with two quality signals from Conversation Intelligence.

  • Task Completion labels each conversation as resolved, unresolved, escalated, or declined by the agent, with a one-sentence reason that cites the moment that drove the decision.
  • Hallucination Detection checks claims your agent makes against the knowledge articles you shared with it and against the rest of the conversation. It returns conflicting claims, the flagged utterance, and what the source says.

Neither signal is enough on its own. A task can be completed and a conversation can end politely while the agent still gives the customer inaccurate information. Looking at both outcome and accuracy helps teams find quality issues earlier and improve how their agents behave over time.

How it works

Task Completion and Hallucination Detection are Twilio-authored language operators in Conversation Intelligence. To use them, add both to an Intelligence Configuration, then add that configuration to your Conversation Configuration in Conversation Orchestrator. When a conversation ends, both operators run on the full conversation. Hallucination Detection checks what your agent said against your Enterprise Knowledge and the conversation itself. Both results go to your web hook and are aggregated by Conversation Insights, where you can query to see patterns.

Each operator uses a large language model as a judge. It reads the conversation, applies the operator’s instructions, and returns a structured result. The judge is separate from the model behind your agent, so your agent isn’t grading its own work. Like any model, a judge can be wrong, so every result comes with a reason or the flagged utterance you can check. Use the results to find conversations worth reviewing, not as a final verdict.

What it looks like in practice

An example below shows how the Task Completion operator labels this conversation resolved as the customer’s question on the refund window was answered, and the Hallucination Detection operator catches the wrong refund window.

Chat flow about a damaged order return, with conflicting return policies flagged.

Find patterns across every conversation

One conversation shows you what went wrong once. Conversation Insights is an API that aggregates the results of both operators across your conversations, so you can see what keeps going wrong. For example, you can count how many conversations were resolved, unresolved, escalated, or declined this week, or see which hallucination types came up most last month. Results are available about 15 minutes after a conversation ends. You can use the API to build your own dashboards and reports, or export the data to the analytics tools you already use.

Get started

This is available to all customers as a public beta. Log in to your Twilio Console to add Task Completion and Hallucination Detection to an Intelligence Configuration, and visit our docs for setup steps and examples. Pricing follows the existing pricing for Twilio-authored Conversation Intelligence operators.


 

Ruma Nair is a Principal Product Manager at Twilio, building AI capabilities for Conversation Intelligence. Her current focus areas include AI agent supervision, real-time intelligence, and evaluations.