Add Natural Language Processing Capabilities to your Twilio Flex Contact Center IVR and Messaging Channels with Google Dialogflow ES

The use of Artificial Intelligence in contact centers has been steadily increasing as the providers' technologies and algorithms become more mature. Today you can find Contact Center AI providers bringing a number of contact center capabilities. These include virtual agents or chatbots capable of understanding natural language, bots with the ability to assist agents while they're having a conversation, and enriched insights with sentiment analytics.

In this post, we will walk you through one way of integrating Twilio Flex with Google Dialogflow ES, one of the Natural Language Processors (NLPs) on the market.

Twilio Flex Dialogflow ES Plugin

This plugin has been built to support both voice and messaging interfaces, with the following capabilities in mind:

• Ability to flatten the IVR tree on a phone call or digital decision tree, as users can use natural language to state the reason for their call. Twilio then routes the interaction based on the intent instead of the traditional DTMF options or chat options.

• Ability to self-serve on any channel. Basic self-service (e.g., answering questions based on an FAQ or knowledge base) and more advanced self-service features (e.g., backend system integrations to enable self-service flows like performing a password reset, ordering a pizza, or scheduling appointments) can be implemented.

• Ability to handover the session to a human agent and provide a transcript of the interaction in the event the virtual assistant is unable to respond to a specific customer intent.

The use cases above, when implemented properly, can contribute to significant savings on contact center operations, improved customer experiences, and additional data insights to why customers are interacting with your contact center.

Prerequisites

• A basic understanding of Google Cloud Platform, Google Dialogflow, Twilio Studio, Twilio Functions and Twilio Flex is recommended but not necessary if you’re willing to learn!
• Comfortable writing code, ideally with some exposure to Node.js and React.
• Twilio CLI to test and deploy the plugin.

Solution components

This solution will use several components of the Twilio stack and Google Dialogflow:

• Twilio Super Network, providing a phone number to reach the contact center via the PSTN,
• Twilio Studio, used to create the call and messaging workflows, will leverage Twilio Functions to integrate with Google Dialogflow and Twilio Sync,
• Twilio Sync, our real-time state synchronization service, will be used to temporarily store interaction transcripts, so the context is not lost in the event of an escalation to a human agent,
• A Twilio Flex component that will invoke Twilio Functions to fetch the transcripts when required and present them on the agent desktop,
• An NLP engine – in this scenario we will use Google Dialogflow ES,
• Twilio Functions, acting as the binding agent for everything else, enabling the integration of all the components listed above.

The diagram below shows how a call will be processed depending on the action captured on Twilio Studio:

Setup

In the next sections we will provide setup instructions on a per component basis.

Google Dialogflow (15 minutes)

Starting with the Google Dialogflow agent, we need to create a new agent and do some initial configuration of intents. To help you get started, please download the following zip file that contains a sample configuration of a Dialogflow agent:

https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/VirtualAgent.zip

Then follow these steps:

1. Navigate to https://dialogflow.cloud.google.com/
2. Login.
3. Click on the cog on the left, and then click on create a new agent.
   
4. Give it a name, set the default time zone and press create.
5. It should take a few seconds to create the agent. Then, click on the cog again, export and import, import from zip.
   
   
   
6. Select the configuration zip file you downloaded, and type IMPORT on the text box. The Google Dialogflow (GDF) import should be successful, and your agent should now look like this when you click on Intents menu on the left:
   
   
   
7. Give it a test by typing car insurance where it says "try now". It should output a response and the Car Insurance Query intent as below:
   
   
   
8. We now have our agent configured with several intents so we are ready to move on. We will need a Google service account to access the agent via API. To set this up, navigate to the general agent options by clicking on the cog once again. Make sure you make a note of your project id, as you will need it later.
   
9. Click on the project ID. This will take you to the Google Cloud Platform application where we will do the next steps. Make sure you have your project ID written somewhere before you move on, as we will need it soon.

Google Cloud Platform (15 minutes)

You've now created the agent on the Dialogflow console and clicked on the project ID to navigate to the GCP console.

What are we doing here? We will need a service account to perform authenticated API requests to Dialogflow. We can create these via the GCP console by following this procedure:

1. Click on the Navigation Menu on the top left, and under "IAM and Admin", click on "Service Accounts".
   
2. Click on the create service account on the top, fill in the form, and then click “Create and Continue”:
   
   
   
3. Click on "Select a role", pick "Dialogflow Service Agent", then press "Continue", and finally press"Done" to complete this process.The account should look something like this:
   
   
   
4. Finally, click on the three dots under "Actions" then "Manage Keys", click on "Add Key" then "Create New Key". Pick a JSON type key and press “Create”. Your browser will download a JSON file automatically.

Make sure you have the JSON file before you move on as we will need it soon.

Twilio Sync (15 minutes)

Now that we have the Google prerequisites sorted, let's work through our Sync requirements. In this scenario, we will be using Twilio Sync to temporarily store conversation transcripts. We need these transcripts if we want to provide context to our agents in the event a virtual assistant is unable to fulfill a customer intent.

To do that, we need to create a Twilio Sync Service and a List. Lets go ahead and do that.

1. Navigate to the Twilio console and log in: https://console.twilio.com/
2. Navigate to Twilio Sync, click "Services", then click "Create a new service".
3. You will be prompted to type a friendly name (I called mine Twilio-transcripts), after you enter one, press "Create".
4. Once you press "Create", you’ll be redirected to a service configuration page. Now, go back to your services page and make note of your Sync Service SID, it starts with ISXXXXX.

At the end of this step you will have created a Sync Service. Make sure you make a note of the SID as we will need it for the next stage.

By the way, there’s nothing stopping you from using a different service to store these transcripts, like an external database. You could potentially use these transcripts to get additional insights on why your customers are calling you. Just don’t forget about GDPR!

Twilio Functions (30 minutes)

Now it’s time to deploy some code! Twilio Functions will be used to bring everything together.

We will need to create functions:

1. A function that will create a transcript on a per call basis,
2. A function that will add new conversations to the transcript,
3. A function that will fetch the contents of the transcript,
4. A function that will be used to invoke the Dialogflow API with the customer message.

We will also need the following information:

• The SID of your Sync Service,
• The SID of your Sync List within the Sync Service,
• The JSON file with the GCP key you downloaded.

Let’s start by creating a Twilio Serverless function and setting up the environment. Navigate to the Twilio Console, then find the Twilio Functions service. Then follow this procedure:

1. Click on create a new service and give it a name, such as Twilio-GDF
2. Click on "Dependencies". First, change your node version to v14. Then add the following dependencies:
   • Twilio-flex-token-validator
   • @google-cloud/dialogflow
3. click on environment variables and add the following:
   • SYNC_SERVICE_SID - <your Sync service SID starting with ISXXXXX>
   • PROJECT_ID - <your dialogflow project ID>

Now, we’re ready to create the functions. Let's start with the function that will create a new transcript on Sync. We can configure a TTL that will automatically delete the transcript once it expires (for GDPR purposes):

1. Click on the "Add" button, then give it the name createNewTranscript
2. Delete the stub code, add the following code, and press "Save":
   https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/Twilio%20Functions/createNewTranscript.js

Now, we need a function to add interactions to the newly created transcript:

1. Click on the add button, then give it the name addToTranscript
2. Delete the stub code, add the following code, and press "Save":
   https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/Twilio%20Functions/addToTranscript.js

We also need a fetch transcript function, that will be used by Twilio Flex to fetch and then present the transcript:

1. Click on the "Add" button, then name the function fetchTranscript
2. Delete the stub code, add the following code, and press "Save":
   https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/Twilio%20Functions/fetchTranscript.js

We need to make this function public, otherwise we won’t be able to use it from Flex. Our function validates requests and will only accept requests with a Twilio Token, so we’re covered. Click where it says "Protected" and then select "Public". Later on, we will need the URL for the fetchTranscript function, so please make a note of it. You can get the URL by clicking on the 3 dots next to the function and clicking on "Copy URL":

One last function is required to integrate with Dialogflow. So far, all you had to do was copy paste code but on this one, you will need to edit and add your GCP credentials.

1. Click on the "Add" button, then name the function fetchTranscript
2. Delete the stub code, add the following code:
   https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/Twilio%20Functions/sendToGDF.js
3. Open the JSON key file you downloaded previously, then copy your private_key value and paste it on the { let private key = ‘ <PKEY>’ } statement highlighted in yellow above. Then, do the same for the {let clientEmail = } statement by adding your client_email value. What we’re doing here is passing the credentials on our request so your function can consume the Dialogflow API. Once you’re done, press "Save".

Now we need to deploy our functions, press "Deploy". Once it finishes deploying, click on "Dependencies" to be sure everything looks like this:

Then, click on "Environment Variables" and it should look like this:

And now we’re done with the serverless functions 😀. On the next step, we’ll be able to see these functions in action!

Twilio Studio call flow (20 minutes)

Now it's time to start putting our test Twilio Studio flow together. We will create a flow based on this template:
https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/Studio%20Templates/Twilio-GDF-Flow-Voice

This flow will start by creating a new transcript for the call, then play a welcome message, then trigger a VoiceCapture widget to capture what the caller says. Then, we invoke the Dialogflow function with the caller's utterances transcribed to text, receive a response from Dialogflow, and write both parts of the interaction into our transcript. We then check for the action within the Dialogflow payload, and if it states escalate, we send the call to the contact center. Otherwise, the call goes back to the the VoiceCapture widget and we loop again and again until the caller hangs up or an escalation action is triggered.

1. Navigate to your Twilio Console and open the Twilio Studio service, then click on "Flows", then click on the plus sign and give it a name, such as Twilio-GDF-Flow-Voice.
2. Now, scroll to the bottom and select the "Import from JSON" option and press "Next"
3. Paste the JSON flow from the template in the link above

We will need to edit our flow now. Click on the CreateNewTranscript widget, then select the Twilio Functions Service we created Twilio-GDF, select the ui environment and the /createNewTranscript function. Then press "Save".

We’ll need to do these changes to most of the remaining nodes as well.

1. Click on the SendToGDF widget, then select the Twilio Functions Service we created Twilio-GDF, select the ui environment and the /sendToGDF function. Then press "Save".
2. Click on the WriteToTranscript, widget, then select the Twilio Functions Service we created Twilio-GDF, select the ui environment and the /addToTranscript function. Then press "Save".
3. Click on the SendToFlex widget, then change the attribute sync_srv_sid to your Sync Service SID starting with ISXXXXX: sync_srv_sid:ISxxxxx

At this point, we are almost ready to test! We need to make sure we have a phone number pointing towards our Studio flow, so make sure you go to the Twilio Console, navigate to "Phone Numbers" -> "Manage" -> "Active Numbers". Then, select a testing number (or purchase a new one) and direct it to our newly created Twilio-GDF-Flow-Voice studio flow.

Test the IVR integration (15 minutes)

You can now call the phone number you just directed to the Studio Flow. When the assistant finishes playing the welcome message, say “hello” and you should hear one of the default intent responses configured on Dialogflow. You can hangup after that.

From the Twilio Console, go into your Twilio Sync Service, then click on "Lists". A new list should have been created, looking something like this:

Navigate to the Dialogflow console, click on your agent, and then click the "Training" option on the navigation bar. You should see the conversations you had with the agent:

If everything worked so far, it means your Studio flow is now integrated with Sync to create and add new information to the transcript. You are also integrated with your Dialogflow Agent.

We've now added a sample flow where if you follow the following script, you will see the awesome potential of the virtual agent:

• Call your phone number and say “I want to renew my home insurance policy”
• When asked for policy number, say “my policy number is 12345”
• When asked for the date of birth, say your date of birth (or any other date)
• You should then hear that the policy has been renewed. You can then say “I’m also looking for car insurance”. The car insurance request will hit the car insurance intent, which has an action of escalation – you will be redirected to Twilio Flex and be placed in queue.

Of course, the insurance policy renewal automation is all made up, with no integration yet developed on Dialogflow. However, if we look at our sync transcript for this conversation you’ll be able to see the whole exchange of messages. You can see how specific parameters are being captured, and you can then use that information either on Dialogflow for intent fulfillment purposes or in Flex to present data to your agents!

It’s time to load up our Flex component so we can present these transcripts to our agents when the call is escalated to the contact center.

Flex Component (15 minutes)

At this point, we already have calls coming in, the integration with Dialogflow in place, and transcripts on Sync. The only thing missing is a Twilio Flex component that presents the transcript to the agent when the agent answers the call.

Let's start by downloading the Twilio Flex component from this link:
https://github.com/rbangueses/Twilio-Dialogflow-Integration/tree/main/Twilio-GDF-Transcript

We’ll need to make one minor change to the code to make it work with our Sync Service. Go into the TranscriptComponent.js (src/components) and put your Twilio function fetchTranscript url on line 22:

 return new Promise((resolve, reject) =>{
   //perform post request and return data
   fetch('<your twilio fetch transcript url>', options)
   .then(response => response.json())
   .then(data => {
     resolve(data);
   })
 })
}

Now, go back to the terminal and type twilio flex:plugins:start to see it in action. Your browser should open a new tab on the Flex Agent desktop – go ahead and log in with your agent.

Now is a good time to give it another test!

Flex transcript testing (15 minutes)

Please follow the following once again:

• Call your phone number and say “I want to renew my home insurance policy”
• When asked for a policy number, say “my policy number is 12345”
• When asked for the date of birth, say your date of birth (or any other date)
• You should then hear that the policy has been renewed. You can then say “I’m also looking for car insurance”. The car insurance request will hit the car insurance intent – this has an action of escalation, so you should be redirected to Twilio Flex and be placed in queue.
• Answer the call on Flex. Notice that you now have a “Transcript’ tab on your task panel. If you click on it, you should see something similar to this:

Enable Dialogflow on Messaging (10 minutes)

To test this on the messaging channels, we will be deploying another Twilio Studio flow. This flow will use the same logic and functions as the Voice Studio flow, so if everything is working for you on Voice, this will be a doodle!

We will create a flow based on this template:
https://github.com/rbangueses/Twilio-Dialogflow-Integration/blob/main/Studio%20Templates/Twilio-GDF-Flow-Messaging

This flow will start by creating a new transcript for the call. Then, send the body of the received SMS to Dialogflow for processing, we receive a response from Dialogflow and we write both parts of the interaction into our transcript. We then check for the action within the Dialogflow payload and if it states escalate, we send the interaction to the contact center. Otherwise, the interaction goes back to send and reply and we loop again and again until the user stops texting or an escalation action is triggered.

1. Navigate to your Twilio Console and open the Twilio Studio service, then click on "Flows", then click on the plus sign and give the new flow a name, such as Twilio-GDF-Flow-Messaging.
2. Now, scroll to the botto, select the "Import from JSON" option, and press Next
3. Paste the JSON flow from the template on the link above
4. Click on the CreateNewTranscript_SMS widget, point it to your createNewTranscript function and press "Save".
5. Click on the SendToGDF_SMS widget, point it to your sendToGDF function and press "Save".
6. Click on the WriteToTranscript_SMS widget, point it to your addToTranscript function and press "Save".
7. Click on the SMS_send_to_flex widget, edit the attributes so then sync_srv_sid has your Sync Service SID value i.e. sync_srv_sid:ISXXXXX
8. Publish your flow

Now, you just need to go to your Twilio phone number and configure it so that it directs the SMS received to your new Studio flow via Flex Proxy Service:

Then pick your Studio flow (make sure the integration type is flex):

Test digital channels (15 minutes)

By now, if everything has been done properly you should be able to have a conversation over SMS with your Dialogflow agent. To confirm everything is working please follow the following script:

• Text your phone number and say “I forgot my password”
• When asked for an account number, say “my account number is 12345”
• When asked for the date of birth, say your date of birth or any other date
• You should then get a response suggesting you’ve been sent an email with a link. Then reply with something like “I want to speak to an agent”
• Flip to your Flex agent desktop and accept the SMS on Flex. Notice that you now have a “Transcript’ tab on your task panel. If you click on it you should see something similar to this:

Come Together

If you managed to get this far, you now have a working virtual assistant deployed on your Twilio environment, capable of handling both voice and sms interactions with some pre-configured intents.

Now you can focus on making it your own! You can create your own Dialogflow intents, implement some flattening of the IVR by looking at the intent names or intent actions, create some self-service journeys, change where you store the transcripts and collect that data for more insightful reporting... the possibilites are endless!

By the way, if you don't have a contact center but still want some of Twilio NLP integration goodness, you can ignore the Flex component element on this article and still have a working solution! Here are a couple of excellent articles that can help you get there through a different path:

• Passing Context from Google Dialogflow to Twilio Studio (Or, There and Back Again)
• Announcing Twilio Programmable Voice One-Click Integration with Google Dialogflow Essentials

Can you think of any more features to add to this? We can't wait to see what you build!