How to Train Bots in Production
This guide will give you the resources and best practices you need to set up tools and processes to continue training your bots after they have been deployed to production and are handling live customer conversations. It answers the following key questions:
- Why should I continue training my bots in production?
- Where can I find the training data?
- What tools does Autopilot provide for on-going training?
To implement the suggestions in this guide, please enable the Log Queries property on your bot. You will not be able to use the REST API or CLI to retrieve past queries from your users if this property is disabled.
Queries
Queries represent words or phrases used by your users when interacting with the bot. By default, Autopilot logs all queries uttered on each channel. They are your primary source of data for training bots in production. Queries can be accessed via the Queries endpoint and viewed in the Queries page in the console.
Training data types
The natural language understanding models used by Autopilot use three types of training data: Samples, Field Values, and Tagged Fields.
Samples
Samples are assigned to Tasks, with each Task requiring training on a corpus of samples for it to be triggered by the bot. As we’ll see below, Queries need to be turned into Samples so that they can be added to the bot’s natural language understanding model. Samples can be accessed via the Samples endpoint on each Task or by clicking into the Task in the console.
Field Values
Values are also part of the training data used to build the natural language understanding model. They represent sample values associated with a Custom Field Type. For example, Magenta
could be a Field Value for a Custom Field Type called Colour
. If your bot is likely to encounter new Field Values, the training process you implement will also need to add them to the model.
Tagged Fields
If your use case requires the bot to extract Fields (both Custom and Built-In) from user utterances, you may also need to tag them in Samples. Field tagging involves specifying which locations in a Sample contain Fields. It involves replacing the Field Value with the unique name of the Field in curly braces in the Sample. For example, here’s how you’d tag Fields called shoe_type
and colour
:
Do you have the new Skechers in red?
Do you have the new {shoe_type} in {colour}?
You learn more about field tagging in the API Reference.
Training tools and processes
The overarching objective of on-going training is to ensure your bot becomes more accurate over time by incorporating data from actual customer conversations into its natural language understanding model. This involves:
- Improving the accuracy of matching intents to Tasks.
- Finding new Field Values to incorporate.
We’ve listed three different ways you can accomplish these tasks below.
1. Queries page
The Queries page in the Autopilot console provides a set of pre-built training tools for incorporating data from real customer conversations into the model.
Improving the accuracy of matching intents to Tasks
- Autopilot Queries can have one of three status values:
- Reviewed for queries that have been used as training data.
- Discarded for queries that don’t need to be used.
- Pending for queries that still need to be reviewed.
- You can filter queries by their status, the tasks they were matched to and the channel. Start by filtering queries with a Pending status.
- Queries that triggered the fallback task will say
no task
in the Task column. - If you find Queries that were matched to the wrong Task, add them as Samples to the appropriate Task. In the example above, “
Do you do buzz cuts?”
should have been matched to theservice_inquiry
Task. - When you assign a Task to this Query, its status gets changed to Reviewed and it is filtered out from your view. It is also automatically created as a Sample.
- Once a Query is marked Reviewed, it can’t be added again as a Sample.
- You can repeat this step periodically as your bot handles more Queries.
Finding new Field Values to incorporate
Let’s say you have a Custom Field called Custom.SERVICE
that helps your bot recognize if the user is inquiring about a particular service:
If they use a new word or phrase for one of these services, or ask about a service that isn’t part of the Custom Field, you should add it to your bot’s training data. You should create a synonym if it has the same meaning as an existing value. You can do this using the Add button on the Query page as shown below:
2. Using the REST API
If you need capabilities beyond those offered by the Queries page in the Console, you can integrate the REST API with existing dashboards, workflows, and other tools, or build a custom training tool.
Improving the accuracy of matching intents to Tasks
1. Retrieve Queries
Start by using the Queries API list resource to retrieve the Queries you need to incorporate into your training data. This endpoint supports filtering by Query status (discarded
, pending-review
and reviewed
) and allows you to retrieve up to 1000 items in one request using the PageSize
query string parameter, similar to Twilio’s default pagination capabilities. Make sure you use the filter to only pull queries that still need review. Each Query has a property called query
that includes the user’s word or phrase and results
that includes the Task matched and Fields detected by the bot, if any.
2. Create Samples
Next, use the Samples API endpoint to add Queries as Samples. You need to make a POST request to this endpoint with the text of the Query included in the TaggedText
parameter.
3. Update Queries
Finally, update the status of the Query you just used to create the Sample to reviewed
so that you only pull new Queries into your training workflow.
Finding new Field Values to incorporate
1. Retrieve Queries
Similar to how you improve the accuracy of matching intents to Tasks, you’ll need to first retrieve Queries using the Queries API list resource.
2. Create Field Values
Next, you’ll need to use the Field Value API endpoint to add the Query as a new Field Value. You need to make a POST request with the text of the Query in the Value
parameter. If the new value is a synonym of an existing one, include it in the SynonymOf
parameter.
3. Update Queries
Finally, update the status of the Query you just used to create the Sample to reviewed
so that you only pull new Queries into your training workflow in the future.
3. Using the CLI
The Autopilot Plugin for the Twilio CLI lets you interact with the API directly via the command line. You can learn how to set up and use it in the docs and in the repo.
Improving the accuracy of matching intents to Tasks
1. Retrieve Queries
You can use the twilio autopilot:queries:export
command to export Queries into a .csv
file containing the following columns, which correspond to properties of the Query API resource. The columns needed for improving the accuracy of matching Tasks are marked in bold:
- Query
- Status
- Source channel
- Date created
- Date updated
- Task unique name
- Field name
- Field value
- Field type
- Language
- AssistantSid
2. Create Samples
You can use the twilio autopilot:samples:upload
command to create Samples. You need to provide a Task SID and a .csv
file containing the Samples to the command. Each Sample should be on its own row in the first column of the file. You can edit the CSV data provided by twilio autopilot:queries:export
to simplify the process.
3. Update Queries
Unfortunately, the Autopilot Plugin for the Twilio CLI does not have a command for updating query status currently. You’ll need to do this directly via the REST API.
Finding new Field Values to incorporate
1. Retrieve Queries
Similar to how you improve the accuracy of matching intents to Tasks, you’ll need to first retrieve Queries using the twilio autopilot:queries:export
command.
2. Create Field Values
You can use the twilio autopilot:fieldvalues:upload
command to create new Field Values. You need to provide the Field Type SID and a .csv
file containing the Field Values to the command. Each Field Value should be on its own row in the first column of the file, with synonyms for each field in subsequent columns. You can edit the CSV data provided by twilio autopilot:queries:export
to simplify the process.
3. Update Queries
Unfortunately, the Autopilot Plugin for the Twilio CLI does not have a command for updating query status currently. You’ll need to do this directly via the REST API.
Building the Model
You must build a new model so that your bot uses the training data you just added when processing queries. There are three ways to build a model with Autopilot currently.
1. Console
You’ll see a notification when Autopilot recognizes that new training data has been added. As a general best practice, build the model after reviewing as many Queries and Fields as possible.
2. API
A POST request to the ModelBuild endpoint will trigger a new model to be built.
3. CLI
The twilio autopilot:modelbuilds:create
command will trigger a new model to be built.
Need some help?
We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd by visiting Twilio's Stack Overflow Collective or browsing the Twilio tag on Stack Overflow.