Use a TwiML Application to Connect your Voice AI Agent to a Twilio Conference

Use a TwiML Application to Connect your Voice AI Agent to a Twilio Conference

As our customers build exciting new Voice AI-backed applications, Twilio is ensuring that our platform provides the capabilities this new class of applications requires while also ensuring that our developer experience remains as strong as ever.

On that note, we are very happy to announce that Twilio now supports adding TwiML Applications to Conferences. Let me explain why this is a big deal.

Many Voice AI-backed applications connect directly to a human caller – either inbound or outbound. In those cases, Twilio enables our customers to use a TwiML <Connect> verb to establish a session with a virtual agent, for example by using ConversationRelay! This works great, but what if an application requires additional functionality like warm or cold transfers, complex recordings, or other specific functionality? The answer is to use Twilio Conference to allow different participants to join and leave the Conference programmatically.

In this post, we’ll show you a proof of concept application where you add a caller and an AI Agent – backed by a TwiML app – to a voice conference, and can then add warm or cold transfers or other features you’ll need to build a world-class voice experience.

Architecture of an AI Agent Twilio Voice conference application

To this point, if our customers have wanted to use the functionality that comes with using Conferences, they have needed to use an additional PSTN call leg to bring in the Voice AI Agent. In addition to being charged at a PSTN rate, it was not easy to pass context to the call leg that connected the Voice AI Agent. There were some workarounds available, but it was not an optimal experience.

To address these issues, Twilio now allows our customers to add participants to Conferences via a TwiML Application. The legs are priced preferably to the PSTN call legs that were required up to this point, and this new call leg option can easily pass context.

Here’s how a call flow could use this new feature:

1. Inbound Caller: The flow starts with an inbound call from a customer. The voice handler for the dialed number triggers the webhook configured by the customer.
2. Add to Conference: The Voice AI Application uses a Twilio Conference so the inbound call is immediately added to a Conference.
3. Add Participant: To connect the AI Agent, the customer’s application calls the Add Participant API where the TO for the participant is set to a TwiML Application for the next instructions. Importantly, the TO field can also include custom parameters (such as a user id, phone number, or other important context).
4. TwiML App: The TwiML App gets invoked and processes the parameters and returns TwiML ( some version of the <Connect> verb) to invoke the session with the AI Agent.
5. AI Agent: The AI Agent is now in the conference as a participant with the necessary context.
6. Options: Since the call is in a Conference, other options like warm or cold transfers are now available.

Prerequisites

• Twilio Account. If you don’t yet have one, you can sign up for a free account here
• A Twilio Phone Number with Voice capabilities
• You can learn to search for and buy a Twilio number here
• Ability to use a command line terminal in your computer, and the ability to set local variables.

Local variables required

To go through the example app build from this post, you will need to get the Twilio Account SID and Twilio Auth Token for the account you wish to use AND you will need them as local variables using the TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN variables:

You can find your Twilio Account SID and Auth Token in your Twilio Console.

Build the AI Agent Conference application

Great! You now have everything you’ll need to build and call the application. Now, let’s walk through the steps you’ll need together.

Step 1: Create a TwiML Bin to create a Conference

The first thing we need to do is create a TwiML Bin to create a conference.

• From the Twilio Console, go to TwiML Bins
• Use the TwiML below this list to create a new TwiML Bin. Use the friendly name simple-conference→ save the name for use later.
• Now go to a Phone Number in your Twilio Account and connect the Voice Configuration -> A call comes in handler for a number to that TwiML Bin.

Step 2: Create a TwiML Bin called PlayCowbell to play a message (voicemail)

We need a second TwiML Bin that is going to play the role of the AI Agent. This bin is simple – it just plays a cowbell.

• From the Twilio Console, go to TwiML Bins
• Use the TwiML below this list that plays a recorded sound and then hangs up.
• Copy the URL of the TwiML Bin once you have saved it. You will need to “click” into the Bin, and the URL should look like this: https://handler.twilio.com/twiml/EHxxxxxxx…

Step 3: Create a TwiML App and connect it to the TwiML Bin

Now we need to create a TwiML Application that will be invoked when we use the Add Participant API. This TwiML Application will point to the TwiML Bin that we created in Step 2.

• From the Twilio Console, go to Voice, then Manage, and then TwiML Apps
• Create a New TwiML App
• Enter the URL from the TwiML Bin created in Step 2 (https://handler.twilio.com/twiml/EHxxxxxxx…) in the REQUEST URL for the Voice Configuration.
• Once saved, click into the new app and copy the TwiML App SID (APxxxxxxx…)

It should look like this in the Console:

Step 4: Call the number connected to the TwiML Bin

We are now ready to test it out! Here is what you should do next:

• Call the number you used in Step 1 from a PSTN device – for example, your cell phone or a landline.
• The call is automatically added to a new conference and then placed on hold until another participant is added.
• You can optionally go to Monitor in your Twilio Console and then select Logs and then Conferences to see the Conference with a single participant.

Put the call on speaker phone and leave the call on hold! (You will hear some lovely hold music.)

In the next step, we’ll manually add an AI Agent to the Conference from the command line.

Step 5: Use the Add Participant API to add the TwiML App to the Conference

Now can make the Add Participant API call to add your AI Agent. Your application would do this programmatically, but we are going to use a terminal window for this demo:

• The curl command below adds the TwiML App as a participant to the conference. Copy the command into a text editor.
• simple-conference is the name of the conference set in Step 1. This can remain unchanged if you used the same name in Step 1.
• app:APxxxxxxx…is the TwiML App SID from Step 3 with the prefix app:. Change the Application SID to the one that you copied in Step 3.
• ?param1=foo&param2=barare parameters that you can pass to the TwiML Application. These are critical – these params allow you to pass context! You can leave these as is, or add your own to test.
• The From number can be any of your Twilio numbers, or it could be an inbound caller phone number if you use the callToken. For this demo, you can use the number that you called.

Once you have edited the command above, paste it into a terminal window and hit return to run it.

Result

If you did everything correctly, you should hear the TwiML app added to the conference and the cowbell sound will play!

You have now successfully connected a TwiML App to a conference! Next, you are ready to point the TwiML App at your Agent – here are some ConversationRelay tutorials to get you started building your own voice agents.

Conference Logs

In the Monitor tab of the Twilio Console, navigate to Logs and then Conferences. You will see the inbound leg and then the outbound call to the application. It should look similar to the image below. Notice the outbound call is to your TwiML application.

Call Logs

Now, again from the Logs section, click Calls and find your Application call to see more details. Notice that it is an incoming call to an Application, and also notice that the parameters you set were passed in and will be available in the event received by your application – pretty awesome!

Conclusion

We are super excited to provide this new functionality, and let you dial a TwiML application into a Conference. We imagine this will enable you to build better Voice AI Applications and more easily connect agents and pass context, without resorting to workaround or extra call legs.

Let us know how you end up using it – we can’t wait to call what you build!.

Dan Bartlett has been building web applications since the first dotcom wave. The core principles from those days remain the same but these days you can build cooler things faster. He can be reached at dbartlett [at] twilio.com.

Margot Hughan is a Product Manager at Twilio and loves working on all things Voice. Her email address is mhughan [at] twilio.com