By 
Everyone has experienced looking up a recipe online then having to scroll through the recipe's back story to reach the end of the page just to find out how to poach chicken. Being an avid cook I began thinking about how a chatbot could take a user request about how to cook a specific food and respond with some quick cooking advice without all the filler.
One challenge of doing so is generating the advice itself. There are a lot of potential cooking questions users could ask and building that knowledge base is an enormous task. When OpenAI released GPT-3 earlier this summer I began to think how GPT-3 could be leveraged to provide cooking advice for any food users ask about. The result is a chatbot that takes user cooking queries and then relies on OpenAI’s GPT-3 language model to generate the content!
In this post I will walk through how Twilio Autopilot and OpenAI together can help provide all kinds of cooking advice. Users will be able to ask their cooking related questions and Autopilot will collect the input and pass it to OpenAI, which will generate a response like the examples in the image below.
Follow along and let's start cooking!
Tutorial requirements
- You will need a Twilio account (this link gets you a free $10 to get started!)
- An OpenAI API key
- Install the Twilio CLI tool and the Autopilot CLI plug-in, which will be used to deploy our Autopilot bot.
Setup the Python virtual environment
The first thing we're going to do is create a virtual environment to work in. For this tutorial we will create a directory called twilio-openai-cooking for our code and other artifacts. To create the project directory and virtual environment for the tutorial open a terminal window and enter the following commands:
mkdir twilio-openai-cooking
cd twilio-openai-cooking
python3 -m venv venv
source venv/bin/activate
pip install openai flask python-dotenv pyngrok
If you are using Windows you can use the following:
md twilio-openai-cooking
cd twilio-openai-cooking
python -m venv venv
venv\Scripts\activate
pip install openai flask python-dotenv pyngrok
Building the Autopilot bot
To interface with our bot we’re going to use Twilio Autopilot. Autopilot helps manage the user conversation by interpreting the user’s request and associating the intent with a task that will fulfill the intent. For our bot the main task is going to listen for cooking related questions. When cooking related questions are identified, Autopilot will route those requests to a Flask endpoint we will create that will make a call to the GPT-3 API to get the cooking instructions. GPT-3 works best when there are prompts to seed the interaction, so having Autopilot manage the conversation so that only cooking related questions trigger the GPT-3 task helps ensure the user input roughly matches the prompts we send to GPT-3 to generate reasonable outputs.
The behavior for a Twilio Autopilot bot is defined in a JSON schema file. The schema file defines the tasks your bot will accomplish, where each task can perform a number of different actions. You may want to check out the Twilio Autopilot Templates GitHub to see a collection of schema templates to get you started on a variety of different use cases!
Within the project directory create a new file called schema.json and populate the file with the following:
{
"friendlyName": "cooking_with_gpt3",
"logQueries": true,
"uniqueName": "cooking_with_gpt3",
"defaults": {
"defaults": {
"assistant_initiation": "task://welcome_message",
"fallback": "task://fallback"
}
},
"styleSheet" : {
"style_sheet" : {
"voice": { "say_voice": "Polly.Matthew" }
}
},
"fieldTypes": [
{
"uniqueName": "Custom.METHOD", "values": [
{ "language": "en-US", "value": "cook", "synonymOf": null },
{ "language": "en-US", "value": "roast", "synonymOf": null },
{ "language": "en-US", "value": "bake", "synonymOf": null },
{ "language": "en-US", "value": "grill", "synonymOf": null },
{ "language": "en-US", "value": "poach", "synonymOf": null }
]
},
{
"uniqueName": "Custom.FOOD", "values": [
{ "language": "en-US", "value": "chicken", "synonymOf": null },
{ "language": "en-US", "value": "eggs", "synonymOf": null },
{ "language": "en-US", "value": "salmon", "synonymOf": null },
{ "language": "en-US", "value": "broccoli", "synonymOf": null },
{ "language": "en-US", "value": "chicken breast", "synonymOf": null },
{ "language": "en-US", "value": "asparagus", "synonymOf": null },
{ "language": "en-US", "value": "zucchini", "synonymOf": null },
{ "language": "en-US", "value": "squash", "synonymOf": null },
{ "language": "en-US", "value": "steak", "synonymOf": null }
]
}
],
"tasks": [
{
"uniqueName": "fallback",
"actions": {
"actions": [
{ "say": "Hmmmm it doesn't look like you have asked about anything cooking related. Please try again." },
{ "listen": true }
]
},
"fields": [],
"samples": []
},
{
"uniqueName": "cooking_tips",
"actions": {
"actions": [
{ "say": "Your cooking advice will be here once we setup our Flask app!" }
]
},
"fields": [
{ "uniqueName": "food", "fieldType": "Custom.FOOD" },
{ "uniqueName": "method", "fieldType": "Custom.METHOD" }
],
"samples": [
{ "language": "en-US", "taggedText": "{method} {food}" },
{ "language": "en-US", "taggedText": "{food}" },
{ "language": "en-US", "taggedText": "how do i {method} {food}" },
{ "language": "en-US", "taggedText": "tell me how to {method} {food}" },
{ "language": "en-US", "taggedText": "how can i {method} {food}" },
{ "language": "en-US", "taggedText": "how long does {food} have to {method}" }
]
},
{
"uniqueName" : "welcome_message",
"actions" : {
"actions" : [
{ "say" : "Hi! Need help cooking? I'm here to help! \n\nYou can ask me things like ' roast asparagus' or 'grill hamburgers'.\n\nTry it out with your favorite foods!" },
{ "listen" : true }
]
},
"fields": [],
"samples": [
{ "language": "en-US", "taggedText": "Hello there" },
{ "language": "en-US", "taggedText": "Whats up" },
{ "language": "en-US", "taggedText": "Heyo" },
{ "language": "en-US", "taggedText": "Hi there" },
{ "language": "en-US", "taggedText": "Yo" },
{ "language": "en-US", "taggedText": "Hey" },
{ "language": "en-US", "taggedText": "How's it going" },
{ "language": "en-US", "taggedText": "Hi" },
{ "language": "en-US", "taggedText": "Hello" }
]
}
],
"modelBuild": { "uniqueName": "V2" }
}
Our bot is named cooking_with_gpt3 defined in the uniqueName field at the top of the schema file. The schema then defines three different tasks the bot will be able to perform:
welcome_messageexplains what the bot does when users enter a greeting like "hello".cooking_tipsis the main task that we'll connect to OpenAI when users ask how to cook certain foods.fallbackis a catch-all for all other user queries that prompts them to ask a proper question.
Each task has a set of actions dictating what should be done when the user triggers the task and a set of sample texts to define what kinds of user queries should be handled by the task.
The schema file also defines two custom fields, Custom.FOOD and Custom.METHOD, to help construct patterns for the task samples. Autopilot has a number of built-in fields you can use to define common data elements in a user’s input, but you can also define your own custom fields like we’re doing here.
The custom fields define common terms and cooking methods that we use to build sample text for the cooking_tips task. Instead of having to provide raw text for all the combinations of cooking methods and foods, we can use these fields to make the samples easier. Looking at the cooking_tips task, there are references to these two fields that are used to help construct our samples.
"samples": [
{ "language": "en-US", "taggedText": "{method} {food}" },
{ "language": "en-US", "taggedText": "{food}" },
{ "language": "en-US", "taggedText": "how do i {method} {food}" },
{ "language": "en-US", "taggedText": "tell me how to {method} {food}" },
{ "language": "en-US", "taggedText": "how can i {method} {food}" },
{ "language": "en-US", "taggedText": "how long does {food} have to {method}" }
]
Installing the Twilio CLI
We will still have to make one more change to the schema file, but for now let’s deploy what we have and we’ll make an update later. The Twilio CLI allows you to manage numerous aspects of your Twilio account and services directly from the command line. One of those services is managing your Autopilot bots. We’re going to use the CLI to create our Autopilot bot.
On a Mac you can install the CLI using Homebrew with the following command:
brew tap twilio/brew && brew install twilio
If you’re using Windows you will have to make sure you have Node.js version 10.12 or above. Check your version by running this command in the command line. If your version is not 10.12 or above you will have to upgrade to a newer version from the Node.js Download page.
node -v
After confirming you have an appropriate version of Node.js enter the following into the command line to install the Twilio CLI:
npm install twilio-cli -g
Once you have installed the CLI you will need to use your Twilio credentials to access your Twilio account through the CLI. You can do this by entering the following in the command line:
twilio login
You will be prompted for your Twilio Account SID and Auth Token. You can find each of these in the Twilio Console in the upper right corner of the Console Dashboard.
Installing the CLI Autopilot Plugin
The last step is to install the Autopilot CLI plugin that will give you access to functionality for interacting with your Autopilot bots. In the command line enter the following command:
twilio plugins:install @dabblelab/plugin-autopilot
You can test the installation was done correctly with this command.
twilio autopilot
You should see a menu of available commands for interacting with Autopilot.
With the CLI tools setup let’s deploy our Autopilot bot!
Deploying the Autopilot bot with the CLI
In a terminal navigate to your project folder and enter the following command:
twilio autopilot:create --schema schema.json
And just like that your Autopilot bot is deployed!
If you make changes to your bot's schema file, updating the bot is easy. Instead of create you can use the update command and your bot will be updated with the new schema.
twilio autopilot:update --schema schema.json
Let’s go practice with our bot to see what it can do so far!
Using the Autopilot Simulator
Twilio provides an online simulator for testing Autopilot bot behaviors. When creating a bot this is a great way to ensure your bot is performing how you would expect.
Log into your Twilio account and select “Autopilot” from the services on the left side menu. Find the cooking_with_gpt3 bot we just created and click on it. From the list of options on the left, select “Simulator”.
From the Simulator you can interact with your bot to see how it behaves. You can try submitting example queries to your bot and see how your bot responds. As you interact with the bot the panel on the right shows you details about how Autopilot interprets each request. This is a great tool for identifying new text samples to trigger your tasks.
You can see how queries related to cooking trigger our cooking_tips task and queries that don’t appear to be cooking related, like building a house, get directed to the fallback task to alert the user that their query isn’t cooking related. Try it out with a few queries of your own!
The panel on the right shows information about the selected task, including the values assigned to the fields in that task.
When you do have queries that trigger the wrong task, you can correct them with the panel on the right. From the dropdown you can select the correct task the request should be routed to then click “Save”. These samples will be added to those tasks on the schema file. Click the “Build Model” button at the bottom of the page to rebuild the model with these new examples. As you provide more samples your bot will improve.
Right now when cooking queries are submitted to our bot the response is some placeholder text. Let’s set up our Flask app to interact with OpenAI to provide more useful responses!
Creating the Flask endpoint to call GPT-3
We need to write the code that will fulfill the cooking_tips task when a user submits a query so we aren’t just responding with the placeholder string. to fulfill that request. In this section we’re going to create a Flask endpoint that will take the user’s cooking query and call the OpenAI API to get the generated recipe advice.
Priming GPT-3
We need to configure a couple of things to get setup to access the OpenAI API. First, we need to put our API key in a secure location. In your project directory create a file named .env. We’re going to store our OpenAI API key in this file. Inside the file include the following line of code with your own API key substituted in.
OPENAI_KEY=your-key
If you're saving your code in a version control system make sure you exclude this .env file to keep your credentials safe!
GPT-3 works best with example prompts to prime the model. The prompts serve as example inputs and outputs for GPT-3 to learn from, so that it can provide relevant responses. For this tutorial the prompts are a series of inputs and outputs where the input is a cooking question and the output provides the answer to the question.
Create the recipes.txt file and input the following text that will serve as our prompts for GPT-3:
input:how do i poach chicken?
output:Place chicken breasts in dutch oven. Cover with cold water seasoned with salt. On medium heat bring water to a boil for 5-10 minutes.
input:how do i cook steak in the oven?
output:Preheat oven to 400 degrees. Allow steaks to reach room temperature. Heat skillet over high heat. Sear prepared steaks for 2 minutes each side. Put in oven for 4 to 5 minutes until cooked to desired temperature.
input:how do i roast broccoli?
output:Preheat oven to 450 degrees. Place broccoli on prepared baking sheet. Roast broccoli on baking sheet for 20 minutes.
input:how do i grill steak?
output:Preheat grill to 450-500 degrees. Allow steak to reach room temperature before putting on grill. Grill 4-5 minutes on one side. then flip to finish cooking...3-5 minutes for Medium Rare (135 degrees), 5-7 minutes for Medium (140 degrees), 8-10 minutes for Medium Well (150 degrees). NOTE: Cooking times may vary based on size of steak.
input:how do i roast asparagus?
output: Preheat oven to 425. Roast asparagus on sheet pan for 12-15 minutes.
input:how do i cook jasmine rice?
output: Use 2 to 1 ratio of water to rice. Bring water to a boil. Add jasmine rice. Cover. Reduce heat to low. Let rice simmer for 15 minutes.
input:how do i grill watermelon?
output:Preheat your grill to medium high heat. Cut the watermelon into 1 inch thick slices. Season with salt and pepper. Cook for 3 minutes on each side. Cut into pieces and serve!
input:how do i roast chicken?
output:Preheat oven to 400 degrees. Allow chicken to reach room temperature. Season with salt and pepper. Place in roasting pan. Roast for 30 to 45 minutes or until internal temperature reaches 165 degrees.
input:how do i bake broccoli?
output:Preheat oven to 425 degrees. Cut broccoli florets into bite size pieces. Place on a roasting pan. Drizzle with olive oil. Season with salt. Bake for 12-14 minutes.
input:grill hot dogs
output:Preheat grill to medium heat. Spray grill with non stick spray. Cook hot dogs for 2 minutes on each side. Cook sausages for 10 minutes on one side.
input:how do i grill ribeye?
output:Preheat grill to high heat. Season the steak with salt and pepper. Grill for 3-4 minutes on each side for medium rare.
Writing the Flask endpoint
Ok, now everything is prepped for us to write the Flask endpoint our cooking_tips task will access. In the project directory create a new file called bot.py that will contain the code for getting the user input from the bot, calling the OpenAI API and then passing the response back to Autopilot. Enter the code below.
import os
import openai
from dotenv import load_dotenv
from flask import Flask, request
app = Flask(__name__)
load_dotenv(".env")
openai.api_key = os.getenv("OPENAI_KEY")
@app.route("/get-cooking-advice", methods=["POST"])
def get_cooking_advice():
# Get the user's input.
user_input = request.values.get("CurrentInput")
# Load the recipe examples.
with open("recipes.txt", "r") as f:
prompt_examples = f.read()
# Build the prompt for the API.
# Combines the examples and appends the user query to the end.
prompt_examples = prompt_examples + f"\n\ninput: {user_input}\n\noutput:"
# Call the OpenAI API.
response = openai.Completion.create(engine="davinci",
prompt=prompt_examples,
max_tokens=200,
temperature=0.1,
frequency_penalty=0.7,
presence_penalty=0.1,
stop=['\n\ninput:', '\ninput', '\noutput:'])
# Build the output response.
if len(response["choices"][0]["text"].strip()) == 0:
msg = "Hmmmmm I don't know that one. Please try again."
else:
msg = response["choices"][0]["text"]
# Make the Autopilot response.
return {
"actions": [
{
"say": msg
}
]
}
if __name__ == "__main__":
app.run()
A few things are happening in this code.
First, we're getting the user query from the request object, which is stored in the CurrentInput field.
# Get the user's input.
user_input = request.values.get("CurrentInput")
Next, the text prompts are loaded from the recipes.txt file into a string variable prompt_examples. Once loaded, the user_input is appended to the end of the string as a new input prompt, and an empty output tag is added to tell GPT-3 that we're looking for output related to the user_input.
# Load the recipe examples.
with open("recipes.txt", "r") as f:
prompt_examples = f.read()
# Build the prompt for the API.
# Combines the examples and appends the user query to the end.
prompt_examples = prompt_examples + f"\n\ninput: {user_input}\n\noutput:"
Now the call to the OpenAI API is made. The prompt_examples are passed as a parameter in the call.
# Call the OpenAI API.
response = openai.Completion.create(engine="davinci",
prompt=prompt_examples,
max_tokens=200,
temperature=0.1,
frequency_penalty=0.7,
presence_penalty=0.1,
stop=['\n\ninput:', '\ninput', '\noutput:'])
There are a number of parameters that can be set to configure the output from OpenAI:
engine: name of the OpenAI engine to use. Must be one of, davinci, ada, babbage, or curie. We’re using davinci here.prompt: example text to feed into the engine (in our case these are example cooking questions / answers)max_tokens: how long the generated response should be. Maximum of 512 characters.temperature: value between 0 and 1 representing the sampling temperature where higher values will result in more creative answers and lower values will produce more well-defined answers. We’re keeping this value lower since we don’t want to get too creative with our cooking responses!frequency_penalty: value between 0 and 1 penalizing the model for repeated tokens, where higher values prevent the model from repeating itself.presence_penalty: value between 0 and 1 penalizing new tokens based on if they appear in the text already. Similar to the temperature parameter we’re keeping this lower to avoid getting too creative with responses.stop: when generating output the API will stop when any of these sequences are met. GPT-3 doesn’t recognize we’re looking for a single response to our query. Having these stop sequences prevents GPT-3 from producing numerousinput:andoutput:blocks where we’re only interested in the initial output. If GPT-3 does generate a newinput:oroutput:block it will not be returned as a part of our output.
Finally, we parse the OpenAI output and generate the response back to the user. In the instance GPT-3 cannot generate output for a given input we will send a message telling the user to try again. Otherwise we will pass the raw text response from OpenAI to the user. The message is sent through the JSON response back to Autopilot.
# Build the output response.
if len(response["choices"][0]["text"].strip()) == 0:
msg = "Hmmmmm I don't know that one. Please try again."
else:
msg = response["choices"][0]["text"]
# Make the Autopilot response.
return {
"actions": [
{
"say": msg
}
]
}
That's all the code! Now we need to make our code accessible to our Autopilot bot.
Setting up Ngrok
The last step is to use ngrok to set up a server that the Autopilot bot can access.
Start the Flask application you just created by typing the following into the terminal:
python3 bot.py
The Flask application is now running locally on your machine. Next we need to make the application visible to the internet so that we can connect it to our Autopilot bot by using ngrok. Open a new terminal window and activate the project’s virtual environment. Then enter the following command:
ngrok http 5000
The output of ngrok will have two lines starting with “Forwarding”. These show the public URL assigned by ngrok, for the http:// and https:// protocols. Any requests sent to these URLs will be automatically forwarded to our Flask application by ngrok.
Updating the schema file
You will see some information about forwarding URLs in the ngrok output. There are two attributes labeled "Forwarding", one is a http address and the other is the https address. Copy the https://....ngrok.io address to your clipboard.
Go back to the schema.json file. The cooking_tips task currently looks like this:
"uniqueName": "cooking_tips",
"actions": {
"actions": [
{
"say": "Your cooking advice will be here once we set up our Flask app!"
}
]
},
Right now the cooking_tips task responds with some text using the say action. Now that we have our Flask endpoint running we have to change the cooking_tips task to use a redirect action. The redirect action will route the user request to the URL or task provided. In this case we’ll be sending the user request to the ngrok endpoint. In the cooking_tips task replace the say action with redirect and enter your ngrok https:// URL followed by /get-cooking-advice, which is the endpoint for the Flask application. The updated schema.json file should look like this now:
"uniqueName": "cooking_tips",
"actions": {
"actions": [
{
"redirect": "https://XXXXXX.ngrok.io/get-cooking-advice"
}
]
},
Save the schema.json file.
Autopilot provides a number of different actions you can perform with your bot beyond just say and redirect. Checkout the Autopilot documentation for more actions you can perform within your bot.
We have locally updated our schema file, but we also have to push the updated schema to Twilio. We can do this using a Twilio CLI command. Enter the following into the command line:
twilio autopilot:update --schema schema.json
Your bot is updated! You can go back to the Simulator and test a few queries to confirm. You should start to see output generated from GPT-3 instead of the placeholder text from before.
All the cooking is done! Now it’s time to do the plating. In the last step we’re going to set up a phone number so we can text our bot to get quick cooking advice on the go!
Purchasing a phone number
To access our bot we need to purchase a phone number. Log into your Twilio account (if you don't have one use this referral link for a free $10 to get started!) From the services menu on the left select “Phone Numbers”. Then choose the “Buy a Number” option. Select a phone number that has SMS capabilities and click “Buy”. Note that if you have a trial account you will be using your trial funds for this purchase.
Connecting Autopilot to our phone number
The last step is to set up your phone number to forward incoming SMS messages to the Autopilot bot you have created. Similar to deploying the Autopilot bot we can update our phone number to forward SMS messages through the Twilio CLI.
To update the phone number we need two pieces of information, your Twilio Account SID and the Autopilot SID for the assistant we created. We can conveniently lookup these pieces of information with the CLI.
You can find your Twilio Account SID by entering the following in the command line:
twilio profiles:list
This will list all the accounts you have configured with the CLI. The second column is the Account SID for each account. These will all have the “AC” suffix. Find the SID for your account and keep it handy.
Next, we have to get the Assistant SID for the Autopilot bot. In the command line you can enter:
twilio autopilot:list
Find the cooking_with_gpt3 bot we created earlier. The first column of the output is the Assistant SID and will have a prefix “UA”. Keep these two SIDs on hand as we’ll need them in the next step.
We have all the information we need to update our phone number to forward SMS messages to our Autopilot bot. All we need to do is issue one more command through the Twilio CLI where you substitute in your Twilio phone number (in E.164 format), your Twilio Account SID, and the Assistant SID you found above.
twilio phone-numbers:update <PHONE_NUMBER> \
--sms-url https://channels.autopilot.twilio.com/v1/<ACCOUNT_SID>/<ASSISTANT_SID>/twilio-messaging
And you're done!
Now send your bot a message about cooking and see the response.
Cooking as easy as G-P-T-3?
GPT-3 is a language model trained to predict the next word in a sequence. GPT-3 doesn't have any actual subject matter knowledge about cooking, it is taking a few input examples (the contents of recipes.txt in this post) and solving a task. So, how accurate is GPT-3 as your sous chef?
For fun I spent a weekend cooking with GPT-3 to test how accurate the advice is. For the most part GPT-3 provided good general advice, but would be a little off on the details. One morning I tried cooking eggs over-easy and the advice provided had all the general steps for how to make the eggs, but the timing was way too long. In other situations like determining how long brown rice cooks the advice was pretty accurate. Given GPT-3 is just a language model with no explicit cooking knowledge the results were pretty good. With some slight modifications to the instructions GPT-3 is a pretty decent sous chef!
Wrapping up
And now it's time to do the dishes! In this tutorial we walked through how the OpenAI API can be incorporated into tasks as a part of an Autopilot bot. Specifically we looked at how OpenAI can help you with quick cooking tips. While I wouldn’t always do as GPT-3 says this was a fun project to work with and I’m excited to see more applications of GPT-3!
Matthew Vielkind is a Senior Data Scientist at CDPHP. Matthew enjoys all things Python and building data products. Follow along with him @MatthewVielkind on Twitter and matty_v_codes on Twitch where he builds new data projects in public sharing what he learns along the way.