How to Train Bots in Production

This guide will give you the resources and best practices you need to set up tools and processes to continue training your bots after they have been deployed to production and are handling live customer conversations. It answers the following key questions:

• Why should I continue training my bots in production?
• Where can I find the training data?
• What tools does Autopilot provide for on-going training?

Queries

Queries represent words or phrases used by your users when interacting with the bot. By default, Autopilot logs all queries uttered on each channel. They are your primary source of data for training bots in production. Queries can be accessed via the Queries endpoint and viewed in the Queries page in the console.

Training data types

The natural language understanding models used by Autopilot use three types of training data: Samples, Field Values, and Tagged Fields.

Samples

Samples are assigned to Tasks, with each Task requiring training on a corpus of samples for it to be triggered by the bot. As we’ll see below, Queries need to be turned into Samples so that they can be added to the bot’s natural language understanding model. Samples can be accessed via the Samples endpoint on each Task or by clicking into the Task in the console.

Field Values

Values are also part of the training data used to build the natural language understanding model. They represent sample values associated with a Custom Field Type. For example, Magenta could be a Field Value for a Custom Field Type called Colour. If your bot is likely to encounter new Field Values, the training process you implement will also need to add them to the model.

Tagged Fields

If your use case requires the bot to extract Fields (both Custom and Built-In) from user utterances, you may also need to tag them in Samples. Field tagging involves specifying which locations in a Sample contain Fields. It involves replacing the Field Value with the unique name of the Field in curly braces in the Sample. For example, here’s how you’d tag Fields called shoe_type and colour :

• Do you have the new Skechers in red?
• Do you have the new {shoe_type} in {colour}?

You learn more about field tagging in the API Reference.

Training tools and processes

The overarching objective of on-going training is to ensure your bot becomes more accurate over time by incorporating data from actual customer conversations into its natural language understanding model. This involves:

1. Improving the accuracy of matching intents to Tasks.
2. Finding new Field Values to incorporate.

We’ve listed three different ways you can accomplish these tasks below.

1. Queries page

The Queries page in the Autopilot console provides a set of pre-built training tools for incorporating data from real customer conversations into the model.

Improving the accuracy of matching intents to Tasks

1. Autopilot Queries can have one of three status values:
   • Reviewed for queries that have been used as training data.
   • Discarded for queries that don’t need to be used.
   • Pending for queries that still need to be reviewed.
2. You can filter queries by their status, the tasks they were matched to and the channel. Start by filtering queries with a Pending status.
3. Queries that triggered the fallback task will say no task in the Task column.
4. If you find Queries that were matched to the wrong Task, add them as Samples to the appropriate Task. In the example above, “Do you do buzz cuts?” should have been matched to the service_inquiry Task.
5. When you assign a Task to this Query, its status gets changed to Reviewed and it is filtered out from your view. It is also automatically created as a Sample.
6. Once a Query is marked Reviewed, it can’t be added again as a Sample.
7. You can repeat this step periodically as your bot handles more Queries.

Finding new Field Values to incorporate

Let’s say you have a Custom Field called Custom.SERVICE that helps your bot recognize if the user is inquiring about a particular service:

If they use a new word or phrase for one of these services, or ask about a service that isn’t part of the Custom Field, you should add it to your bot’s training data. You should create a synonym if it has the same meaning as an existing value. You can do this using the Add button on the Query page as shown below:

2. Using the REST API

If you need capabilities beyond those offered by the Queries page in the Console, you can integrate the REST API with existing dashboards, workflows, and other tools, or build a custom training tool.

Improving the accuracy of matching intents to Tasks

1. Retrieve Queries

Start by using the Queries API list resource to retrieve the Queries you need to incorporate into your training data. This endpoint supports filtering by Query status (discarded, pending-review and reviewed) and allows you to retrieve up to 1000 items in one request using the PageSize query string parameter, similar to Twilio’s default pagination capabilities. Make sure you use the filter to only pull queries that still need review. Each Query has a property called query that includes the user’s word or phrase and results that includes the Task matched and Fields detected by the bot, if any.

2. Create Samples

Next, use the Samples API endpoint to add Queries as Samples. You need to make a POST request to this endpoint with the text of the Query included in the TaggedText parameter.

3. Update Queries

Finally, update the status of the Query you just used to create the Sample to reviewed so that you only pull new Queries into your training workflow.

Finding new Field Values to incorporate

1. Retrieve Queries

Similar to how you improve the accuracy of matching intents to Tasks, you’ll need to first retrieve Queries using the Queries API list resource.

2. Create Field Values

Next, you’ll need to use the Field Value API endpoint to add the Query as a new Field Value. You need to make a POST request with the text of the Query in the Value parameter. If the new value is a synonym of an existing one, include it in the SynonymOf parameter.

3. Update Queries

Finally, update the status of the Query you just used to create the Sample to reviewed so that you only pull new Queries into your training workflow in the future.

3. Using the CLI

The Autopilot Plugin for the Twilio CLI lets you interact with the API directly via the command line. You can learn how to set up and use it in the docs and in the repo.

Improving the accuracy of matching intents to Tasks

1. Retrieve Queries

You can use the twilio autopilot:queries:export command to export Queries into a .csv file containing the following columns, which correspond to properties of the Query API resource. The columns needed for improving the accuracy of matching Tasks are marked in bold:

• Query
• Status
• Source channel
• Date created
• Date updated
• Task unique name
• Field name
• Field value
• Field type
• Language
• AssistantSid

2. Create Samples

You can use the twilio autopilot:samples:upload command to create Samples. You need to provide a Task SID and a .csv file containing the Samples to the command. Each Sample should be on its own row in the first column of the file. You can edit the CSV data provided by twilio autopilot:queries:export to simplify the process.

3. Update Queries

Unfortunately, the Autopilot Plugin for the Twilio CLI does not have a command for updating query status currently. You’ll need to do this directly via the REST API.

Finding new Field Values to incorporate

1. Retrieve Queries

Similar to how you improve the accuracy of matching intents to Tasks, you’ll need to first retrieve Queries using the twilio autopilot:queries:export command.

2. Create Field Values

You can use the twilio autopilot:fieldvalues:upload command to create new Field Values. You need to provide the Field Type SID and a .csv file containing the Field Values to the command. Each Field Value should be on its own row in the first column of the file, with synonyms for each field in subsequent columns. You can edit the CSV data provided by twilio autopilot:queries:export to simplify the process.

3. Update Queries

Unfortunately, the Autopilot Plugin for the Twilio CLI does not have a command for updating query status currently. You’ll need to do this directly via the REST API.

Building the Model

You must build a new model so that your bot uses the training data you just added when processing queries. There are three ways to build a model with Autopilot currently.

1. Console

You’ll see a notification when Autopilot recognizes that new training data has been added. As a general best practice, build the model after reviewing as many Queries and Fields as possible.

2. API

A POST request to the ModelBuild endpoint will trigger a new model to be built.

3. CLI

The twilio autopilot:modelbuilds:create command will trigger a new model to be built.