Add Function and Tool Calling to a Twilio Voice OpenAI Integration
Time to read: 6 minutes
Add Function and Tool Calling to a Twilio Voice OpenAI Integration
ConversationRelay is a product from Twilio that allows you to build real-time, human-friendly voice applications. With ConversationRelay, you can interact with your AI Large Language Model (LLM) of choice. It opens a WebSocket so you can integrate with any AI API, allowing for a fluid, event-based interaction over a fast two-way connection.
In previous tutorials, we talked about getting set up with ConversationRelay and getting your AI application running integrating OpenAI. After getting started with ConversationRelay and OpenAI, the next tutorial updated our application for better interruption handling using Token Streaming.
What's next for our ConversationRelay application? In this chapter of this tutorial series, you will learn how to add Tool Calling to your application. This update will allow you to integrate your AI conversation with additional APIs of your choice, using the example of calling a simple joke retrieval API, jokeapi.dev.
Tool Calling can open up a world of possibilities for your application, allowing you to interact with calendars, databases, or whatever you can imagine. Let's get to the code!
Prerequisites
To deploy this tutorial you will need:
- Node.js installed on your machine
- A Twilio phone number (Sign up for Twilio here)
- Your IDE of choice (such as Visual Studio Code)
- The ngrok tunneling service (or other tunneling service)
- An OpenAI Account to generate an API Key
- A phone to place your outgoing call to Twilio
Write the code
This tutorial is a sequel to our ConversationRelay Quickstart and Better Interruption Handling tutorials in Node.js. For this tutorial you have two options: you can either use a pre-existing tutorial, cloned from this repo, or you can start your application from scratch. Instructions for both are provided here.
If you're starting this tutorial from scratch, start by creating a new folder for your project.
Next, initiate a new node.js project, and install the prerequisites.
If you are, instead, building on the tutorial from our previous post, there's only two new things you have to install:
In this build you are using Fastify as your framework. It lets you quickly spin up a server for both the WebSocket you'll need, as well as the route for the instructions you're going to need to provide to Twilio.
One new addition here from previous tutorials is adding @fastify/formbody
. The extension will allow your application to process the body of the response from your external API tool, allowing you to POST
to the endpoint from your Twilio webhook (the default in the Console). This line also adds axios
, a JavaScript library for making HTTP requests. This will be used to communicate with your tool API.
Creating the Environment File
This build will also – as before, if you completed the other steps – require an API key for OpenAI.
If you try to expand this tutorial to call other APIs in the future, you will also need to store keys for those APIs in a safe place. Therefore, it’s best practice to have a project .env file. If you do not have one, create it now in your project folder.
In .env, use the following line of code, replacing the placeholder shown with your actual key from the OpenAI API keys page.
You'll add more to this environment file as you work through the project.
Write your server code
If you're building off the old quickstart, you should have a file called server.js. If you don't have this file yet, create it now, in the same folder as your .env file. This is where the primary code for your project server is going to be stored. Create this file in the same directory as your .env file.
At the top of the file are all the necessary imports.
This code will go immediately beneath the above. This is an alteration from previous tutorial code.
Here, you're going to alter the WELCOME_GREETING
prompt slightly from previous tutorials just to let users know what specific functionality your assistant has. You are then changing the SYSTEM_PROMPT
to make sure that the OpenAI API understands when it's the appropriate time to access the API. The get_programming_joke
function will be called when the user is asking for a programming joke, and this demonstrates the tool calling with the additional API.
Adding the getJoke Function
Next, we're adding this getJoke
function which allows us to use the jokeapi.dev service when a user requests a programming joke. This is a simple API that does not require any external authentication. For any API you want to call in the future, you might need to add additional API keys, parameters, or security.
Add this code to your server.js.
Setting up Server Configuration
This next block of code will be added below the getJoke
function. This code sets up your server to handle the calls. It's unchanged from the previous tutorial, but replicated here to make it easier to follow this tutorial. If you're using the old server.js file, you can leave this as written.
Interruption Handling
This is the code we used to handle interruptions in the previous tutorial. That code is better explained there. Since there is no need to remove this functionality from the updated conversation bot, this code is left in.
Use this code to finalize the bot. It handles interruptions and finishes creating your server.
Run and test
It's time to test your application. If you've followed previous tutorials, these steps should be familiar. Start by using ngrok to open a tunnel to your locally running application.
You will need to keep the ngrok url for use in two places: in the Twilio console, and in your environment files.
Get the URL for your file and add it to the .env file using this line:
Replace the beginning of this placeholder with the correct information from your ngrok url. Note that you do not include the scheme (the “https://” or “http://”) in the environment variable.
Now you are ready to run your server. Return to your code and type the following into the console:
Go into your Twilio console, and look for the phone number that you registered.
Set the configuration under A call comes in with the Webhook option as shown below.
In the URL space, add your ngrok URL (this time including the “https://”), and follow that up with /twiml
for the correct routing. Everything else can be left as default, including HTTP POST.


When a call is connected, Twilio will first speak the greeting message that you provided. Then it will use the provided ngrok URL to connect directly to the websocket. That websocket connection will open up the line for you to have a conversation with OpenAI.
Save your configurations in the console. Now dial up the number on your phone.
If everything is hooked up correctly, you should hear your customized greeting.
To test the tool calling portion of your application, try asking the AI assistant to respond with a joke about programming. This interaction should call the appropriate API and retrieve that for the body. Then the programming joke it retrieves will be included in your response!


What else can you do with ConversationRelay?
Thank you for reading our series on ConversationRelay. With these tutorials you have learned how to:
- Set up an application for real time phone conversations with an AI assistant
- Use Token Streaming to decrease latency with the AI conversation
- Interrupt your AI conversation and allow the AI to have knowledge about where that interruption occurred
- Call Tools that allow the AI to interact with different APIs.
Imagine the possibilities you can open up building with ConversationRelay! Let's build something amazing that helps you be more productive and have fun too.
My colleagues have built some awesome sample applications and demos on top of ConversationRelay. Here’s a few more articles that you can check out:
- ConversationRelay Architecture for Voice AI Applications Built on AWS
- VoiceAI: Building Voice Bots with Twilio's ConversationRelay
- Integrate Twilio ConversationRelay with Twilio Flex for Contextual Escalations
- Twilio Agentic Voice Assistant built on ConversationRelay
Amanda Lange is a .NET Engineer of Technical Content. She is here to teach how to create great things using C# and .NET programming. She can be reached at amlange [ at] twilio.com.
Related Posts
Related Resources
Twilio Docs
From APIs to SDKs to sample apps
API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
Resource Center
The latest ebooks, industry reports, and webinars
Learn from customer engagement experts to improve your own communication.
Ahoy
Twilio's developer community hub
Best practices, code samples, and inspiration to build communications and digital engagement experiences.