How to Connect Your Twilio Agent to External APIs in Python

Time to read:

September 12, 2026

Written by

Twilion

How to Connect Your Twilio Agent to External APIs in Python

The world is starting to increasingly rely on voice-enabled AI agents to get work done. But an agent can only do so much. Voice AI agents by themselves are able to hold conversations, but what happens if the agent needs to do something like retrieve customer data, look at an inventory, or book an appointment for a user?

In order to make your AI agent truly helpful, you need for that AI agent to have access to real time information. External APIs can provide that information. When your agent works together with an API, your agent is empowered to get information your users really need, and take actions on the user’s behalf like viewing inventory, calendars, menus, and more.

In this tutorial, you will use Python and FastAPI to build a voice agent using Twilio Conversation Relay. Your agent will use tool calling from an LLM-driven conversation to dynamically fetch live data from an external REST API. This tutorial uses a simple API with no additional authentication requirements to showcase the potential of the AI tool. When you have completed the tutorial, you should understand the pipeline to interact with an external API, and how you could employ this functionality in your own builds.

Prerequisites

To complete this tutorial you will need the following:

A free Twilio account with a voice-capable phone number
Python 3.11 or later
An OpenAI API key
ngrok to expose local webhooks to Twilio
An IDE or text editor such as Visual Studio Code

Building the application

Step 1 - Set up the FastAPI project

Your first step is creating a new folder and a Python virtual environment. Go into your terminal and type the following:

mkdir twilio-agent-api
cd twilio-agent-api
python -m venv .venv
source .venv/bin/activate

If you are on Windows, activate your virtual environment with .venv\Scripts\activate instead. A virtual environment keeps the packages you install for this project separate from the rest of your system, which helps avoid version conflicts with other Python projects.

Step 2 - Install dependencies

Install the packages you will need for your project by typing the following into your terminal:

pip install fastapi "uvicorn[standard]" openai python-dotenv twilio httpx

These packages are necessary for your project setup: FastAPI and uvicorn will run your web server and handle the websocket connection. The openai package will be used to connect your solution to OpenAI. The twilio package will allow your application to generate TwiML. The python-dotenv package allows you to import your environment variables into your solution using a .env file, and httpx will let your application make asynchronous requests to the external API.

Step 3 - Configure environment variables

This tutorial is simple enough not to require much information from your Twilio account. But you will need somewhere to safely store your OpenAI API key. Create a file called .env in your project folder. Add to that file the following text:

OPENAI_API_KEY=VAxxxxxxxxxxx

Your OpenAI API Key is generated from OpenAI’s dashboard. You shouldn’t need any other keys in this file. However, if you decide later to call an API that has additional authentication, that key can be stored here as well.

Step 4 - Build the base application

You will use the very simple API, Cat Facts, in this demo. Our application is going to make a simple API call to request a “Cat Fact” from our agent. This API requires no additional authentication and has simple output, which makes it very useful for a demonstration.

Create a new file in your project folder called main.py. Open this file in your IDE of choice, and adjust it to have the following code:

import os
from dotenv import load_dotenv
from fastapi import FastAPI, Request
from fastapi.responses import Response
from openai import AsyncOpenAI
from twilio.twiml.voice_response import Connect, VoiceResponse
load_dotenv()
app = FastAPI()
client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
@app.post("/voice")
async def voice_endpoint(request: Request):
    host = request.headers.get("host")
    response = VoiceResponse()
    connect = Connect()
    connect.conversation_relay(
        url=f"wss://{host}/ws",
        welcome_greeting="Hello! Ask me for a cat fact.",
    )
    response.append(connect)
    return Response(content=str(response), media_type="text/xml")

This code is making a connection to a websocket to enable your agent. You are using Conversation Relay to build the connection between OpenAI and your voice-capable Twilio number, creating a voice agent that can hold a natural sounding conversation. The host is read straight off the incoming request’s Host header, so the websocket URL always matches whatever ngrok hostname is currently forwarding to your server, no manual configuration needed. Notice that you have also added a simple greeting for your agent using the welcome_greeting parameter, which is generated into TwiML by the Twilio helper library. This greeting line can be adjusted as needed to give the user an initial prompt for interaction.

Step 5 - Handle Conversation Relay

You will need some additional code to connect your websocket to Conversation Relay. Add the following to the bottom of your main.py file:

import json
from fastapi import WebSocket, WebSocketDisconnect
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    call_sid = None
    messages: list[dict] = []
    try:
        while True:
            raw = await websocket.receive_text()
            data = json.loads(raw)
            msg_type = data.get("type")
            if msg_type == "setup":
                call_sid = data.get("callSid")
                print(f"[{call_sid}] Call connected")
            elif msg_type == "prompt" and data.get("last"):
                user_text = data.get("voicePrompt", "")
                print(f"[{call_sid}] Caller: {user_text}")
                messages.append({"role": "user", "content": user_text})
                response_text = await stream_response(websocket, call_sid, messages)
                messages.append({"role": "assistant", "content": response_text})
            elif msg_type == "interrupt":
                spoken = data.get("utteranceUntilInterrupt", "")
                print(f"[{call_sid}] Interrupted after: '{spoken}'")
                if messages and messages[-1]["role"] == "assistant":
                    messages.pop()
            elif msg_type == "error":
                description = data.get("description")
                print(f"[{call_sid}] Conversation Relay error: {description}")
    except WebSocketDisconnect:
        print(f"[{call_sid}] Call ended")

This code is communicating with your websocket, breaking your voice inquiries down into conversation messages to be processed by the AI. Changing your voice responses to text, it then streams that text in real time to the AI in order to get fast responses.

This is one important component, but you still need to make the connection to OpenAI. You will do that in the next step.

Step 6 - Connect to OpenAI

In this step, you will configure a tool function schema using the OpenAI Python SDK, and write the function that streams responses back to the caller.

Write the system prompt instructing the agent when to execute external API calls based on user voice prompts. You’ll see the prompt inside the SYSTEM_PROMPT constant in the code below. You can adjust this to your needs. In this prompt, you make sure that the AI realizes it’s being used for voice interaction, by reminding it not to use any bullet points or emojis when it communicates.

Add the following to the top of your main.py file, just below your other imports:

import httpx
MODEL = "gpt-4o-mini"
SYSTEM_PROMPT = """
You are a cat fact generator. If your user asks you for a cat fact you will respond with a cat fact.
Speak naturally as if talking on the phone. Use plain sentences only. Do not use lists or bullet points. Do not use any emojis.
When you are asked for a cat fact you must call the get_cat_fact tool to retrieve one from the API.
Do not just make up facts. If you do not know the answer, respond with "I don't know."
"""
CAT_FACT_TOOL = {
    "type": "function",
    "function": {
        "name": "get_cat_fact",
        "description": "Retrieves a random cat fact from the catfact.ninja API.",
        "parameters": {
            "type": "object",
            "properties": {},
            "required": [],
        },
    },
}
async def get_cat_fact() -> str:
    try:
        async with httpx.AsyncClient() as http_client:
            response = await http_client.get("https://catfact.ninja/fact")
            response.raise_for_status()
            data = response.json()
            return data.get("fact", "No fact returned.")
    except httpx.HTTPError as exc:
        return f"Error retrieving cat fact: {exc}"
async def execute_tool(name: str) -> str:
    if name != "get_cat_fact":
        return "Unknown tool."
    return await get_cat_fact()

The get_cat_fact function is what actually calls our external API. It reaches out to the API located at https://catfact.ninja and parses the json response from the API. If it can’t find a fact, say, if the connection to the API is interrupted, it returns an error.

Now add the streaming function that ties the system prompt, the tool, and the OpenAI SDK together. Add this to the bottom of your main.py file:

async def stream_response(
    websocket: WebSocket,
    call_sid: str | None,
    messages: list[dict],
) -> str:
    full_response = ""
    openai_messages = [{"role": "system", "content": SYSTEM_PROMPT}] + messages
    stream = await client.chat.completions.create(
        model=MODEL,
        max_tokens=300,
        messages=openai_messages,
        tools=[CAT_FACT_TOOL],
        stream=True,
    )
    finish_reason = None
    tool_calls_acc: dict[int, dict] = {}
    async for chunk in stream:
        choice = chunk.choices[0]
        delta = choice.delta
        finish_reason = choice.finish_reason or finish_reason
        if delta.content:
            full_response += delta.content
            await websocket.send_text(
                json.dumps({"type": "text", "token": delta.content, "last": False})
            )
        if delta.tool_calls:
            for tc in delta.tool_calls:
                entry = tool_calls_acc.setdefault(
                    tc.index, {"id": "", "name": "", "arguments": ""}
                )
                if tc.id:
                    entry["id"] = tc.id
                if tc.function and tc.function.name:
                    entry["name"] = tc.function.name
                if tc.function and tc.function.arguments:
                    entry["arguments"] += tc.function.arguments
    try:
        if finish_reason == "tool_calls" and tool_calls_acc:
            tool_calls = list(tool_calls_acc.values())
            openai_messages.append(
                {
                    "role": "assistant",
                    "tool_calls": [
                        {
                            "id": tc["id"],
                            "type": "function",
                            "function": {
                                "name": tc["name"],
                                "arguments": tc["arguments"] or "{}",
                            },
                        }
                        for tc in tool_calls
                    ],
                }
            )
            for tc in tool_calls:
                result = await execute_tool(tc["name"])
                print(f"[{call_sid}] Tool {tc['name']}() -> {result}")
                openai_messages.append(
                    {
                        "role": "tool",
                        "tool_call_id": tc["id"],
                        "content": result,
                    }
                )
            full_response = ""
            stream2 = await client.chat.completions.create(
                model=MODEL,
                max_tokens=400,
                messages=openai_messages,
                stream=True,
            )
            async for chunk in stream2:
                delta = chunk.choices[0].delta
                if delta.content:
                    full_response += delta.content
                    await websocket.send_text(
                        json.dumps(
                            {"type": "text", "token": delta.content, "last": False}
                        )
                    )
    finally:
        await websocket.send_text(json.dumps({"type": "text", "token": "", "last": True}))
    print(f"[{call_sid}] Assistant: {full_response}")
    return full_response

This function streams tokens from OpenAI back to your websocket as they are generated, which keeps the perceived latency low for the caller. If the model decides it needs a cat fact, it responds with a tool call instead of text. Your code then executes that tool, sends the result back to OpenAI, and streams the model’s follow-up response, the one that actually answers the caller, back over the websocket.

Testing your application

Now it is time to test your application and chat with your AI.

First, run your application using this command in the terminal:

uvicorn main:app --reload

Once your webhook is running, you will need to expose it to the internet by using ngrok or another tunneling service.

ngrok http 8000

Replace 8000 with whatever port your application is running on if you have a different port.

Now ngrok will provide you with a url for utilizing in your Twilio console. Go into your Twilio console and find the Twilio phone number that you prepared. Under the option A Call Comes In, choose Webhook, and fill in your ngrok URL followed by /voice, as shown in the graphic below:

configuration for webhooks in the Twilio console

Be sure also that your HTTP block is set to POST.

Now save this configuration, and call your Twilio Phone Number.

You should hear a message with the AI greeting that you provided in main.py.

Try asking your AI about a cat fact and you will get a cat fact from the cat fact API!

Troubleshooting

If you are having some difficulty with your call, there are some common problems you might want to check. First of all, make sure your ngrok URL is correct in the console and matches the one that’s in your terminal, with /voice appended to the end.

If you are still having issues, check your environment variables. You will need to make sure your API keys are correct for any key that you happen to be using, including your key for OpenAI. The sample API requires no additional keys, but if you decide to expand the application, you will also need to authenticate any external APIs that you call. Check the rules for your individual APIs.

Conclusion

Connecting LLM function tools to external HTTP endpoints empowers Twilio voice agents with real-time data. With the use of external APIs, you can create an agent that doesn’t just respond to questions, but truly does the work your customers need.

Are you looking for some further project ideas or further reading? We also have a series on getting started creating your AI Phone Agent with Conversation Relay.

We can’t wait to see what we can help you build!

Related Resources

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.

How to Connect Your Twilio Agent to External APIs in Python

Related Posts

Related Resources