Contact sales Start for free
Contact sales Start for free

How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and Python

Time to read:

September 10, 2026
Written by
Xinghao Huang
Twilion
Reviewed by
Ryan Rouleau
Twilion
Ryan Rishi
Twilion
Rachel Baskin
Twilion
Wen Zhu
Twilion
Paul Kamp
Twilion

How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and Python

"It works great in my testing script, but the second we put it on a real phone call, it falls apart." That's a common complaint once a voice AI demo meets production traffic.

OpenAI's GPT-Live is a family of full-duplex speech-to-speech, or S2S, models that cleanly separates having the conversation from doing the work: tool calls are delegated to your own backend or an OpenAI-hosted Responses API model, instead of being baked into the loop that's also managing speech.

Twilio Agent Connect, Twilio's open-source Python SDK for building voice and messaging AI agents, now ships a GPTLiveProvider that bridges Twilio Programmable Voice Media Streams directly to GPT-Live-1 in the OpenAI API through a flexible, declarative config interface, with no custom WebSocket plumbing required.

In this tutorial, you'll use Python and build a phone number that, when called, connects inbound callers to a GPT-Live-powered voice agent, has it greet the caller proactively, and asks it a question that triggers a tool call. Then, as a further example, you'll place an outbound call with Twilio and GPT-Live. I’ll also show you some other features like per-call customization, and you can explore from there. Let’s get started!

Prerequisites and common pitfalls to getting started

A couple of things make this integration simpler than a typical speech-to-speech integration:

  • GPT-Live is full-duplex and handles interruption logic. There's no barge-in or audio-truncate bookkeeping to write on your side. Compare that to a half-duplex model, where the app has to detect when the caller interrupts and tell the model to stop.
  • GPT-Live's audio format can be configured to natively match Twilio Media Streams' wire format ( 8kHz mu-law).

Build the app

The full working example lives at openai_gpt_live.py. Follow along below, or clone the repo and run that file directly.

Step 1: Set up TAC and the FastAPI server

Every TAC voice app starts with a Twilio Agent Connect instance and a VoiceChannel. VoiceChannel is provider-agnostic: the same class hosts Conversation Relay, GPT Realtime, and GPT-Live, depending on which config you pass it.

from dotenv import load_dotenv
from tac import TAC, TACConfig
from tac.channels.voice import VoiceChannel
from tac.channels.voice.media_streams.gpt_live import (
    TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE,
    GPTLiveProviderConfig,
)
from tac.server import TACFastAPIServer
load_dotenv()
tac = TAC(config=TACConfig.from_env())

TACConfig.from_env() reads its settings from environment variables.

Create a .env file in your project root with TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN (found on your Console dashboard), TWILIO_API_KEY and TWILIO_API_SECRET (from API keys & tokens), TWILIO_PHONE_NUMBER (from Phone Numbers > Manage > Active Numbers), TWILIO_VOICE_PUBLIC_DOMAIN (your ngrok domain from the prerequisites step), and OPENAI_API_KEY, and load_dotenv() above picks them up automatically.

Step 2: Configure the GPT-Live session and greeting

default_session_config is sent as GPT-Live's session.start payload the moment the model connects. At minimum it needs instructions (the system prompt) and audio.format set to TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE. 

DEFAULT_SESSION_CONFIG = {
    "instructions": (
        "You are a warm, friendly voice assistant speaking with a caller over the phone. "
        "Keep responses short — a sentence or two per turn."
    ),
    "audio": {
        "format": TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE,
        "output": {"voice": "marin"},
    },
}
voice_channel = VoiceChannel(
    tac,
    config=GPTLiveProviderConfig(
        default_session_config=DEFAULT_SESSION_CONFIG,
        welcome_instruction=(
            "Greet the caller immediately using the exact text below. Do not wait "
            "for the caller to speak first. After the greeting, pause and listen."
            "\n\nHello! You've reached a voice agent powered by Twilio and OpenAI's "
            "GPT-Live. How can I help you today?"
        ),
    ),
)

Setting welcome_instruction makes the agent speak first instead of waiting for the caller, which is useful for any call where the caller doesn't know they've reached an AI agent yet.

Step 3: Wire up a tool call

GPT-Live doesn't call functions directly. It delegates tool calls either through Client Delegation (where you need to build the delegation integration) or through a Responses API model. In this tutorial, we’ll use Responses delegation.

Any @function_tool-decorated Python function can be exposed this way: register it in tools=[...] so Twilio Agent Connect can execute it, and separately list its schema under delegation.responses.tools so the model knows it exists. 

from tac.tools import function_tool
@function_tool()
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    time.sleep(3)  # simulate a slow backend call
    return f"It's sunny and 72F in {city}."
DEFAULT_SESSION_CONFIG["delegation"] = {
    "type": "responses",
    "responses": {
        "model": "gpt-5.6-sol",
        "tools": [get_weather.to_realtime_format()],
        "tool_choice": "auto",
    },
}
voice_channel = VoiceChannel(
    tac,
    config=GPTLiveProviderConfig(
        tools=[get_weather],
        default_session_config=DEFAULT_SESSION_CONFIG,
        welcome_instruction=(
            "Greet the caller immediately using the exact text below. Do not wait "
            "for the caller to speak first. After the greeting, pause and listen."
            "\n\nHello! You've reached a voice agent powered by Twilio and OpenAI's "
            "GPT-Live. How can I help you today?"
        ),
    ),
)

Keeping tool execution outside the real-time voice loop means your tool logic can be as slow or complex as it needs, whether that's a database lookup, an API call, or a business rule, without adding latency to the conversation. (The time.sleep(3) above just makes that visible: run the demo and the call keeps going while the tool runs.) GPT-Live delegates the decision to call a tool to a Responses API model, then Twilio Agent Connect runs the actual Python function and speaks back the result.

And hopefully, that shows how straightforward OpenAI’s Responses delegation is – when you build your agent, you’ll swap get_weather for a function that hits your own backend, for example looking up an order status, checking appointment availability, or pulling account details. All the same wiring applies without changing anything else here.

Step 4: Wire up the server 

server = TACFastAPIServer(tac=tac, voice_channel=voice_channel, app=None)
server.start()

TACFastAPIServer mounts the TwiML endpoint at /twiml by default.

Step 5: Point your Twilio number at the webhook

With the server running behind a public tunnel (like ngrok), tell Twilio where to send incoming calls:

  • In the Twilio Console, go to Phone Numbers > Manage > Active Numbers and select your number.
  • Under Voice Configuration, set "A call comes in" to Webhook, and enter https://<your-domain>/twiml.
  • Set the HTTP method to POST, then save.

Now, you're ready to call in! ☎️

And there you have it: you now have a phone number that greets callers, answers questions, and calls real functions, with no WebSocket code required.

Run, test, troubleshoot, or product demonstration

Call your Twilio number… if everything is set up correctly, you should hear the agent's welcome_instruction immediately, without saying anything first. After the greeting, ask "what's the weather in Los Angeles?" — the model should invoke get_weather via Responses delegation and read back the result in the same turn.

Have fun talking to your agent – and when you hang up, the transcript accumulated in ConversationSession.metadata["transcript"] prints to your console if you register an on_conversation_ended callback that reads it off the session when the call ends. Here’s an example: 

@tac.on_conversation_ended
async def handle_conversation_ended(context: ConversationSession) -> None:
    transcript = context.metadata.get("transcript", [])
    print(f"Call {context.conversation_id} ended. Transcript:")
    for turn in transcript:
        print(f"  {turn['role']}: {turn['text']}")
Connected to GPT-Live [conversation_id=CA_demo123]
Tool call: get_weather({"city":"Los Angeles"})
Tool result: get_weather -> It's sunny and 72F in Los Angeles.
Media stream stopped [conversation_id=CA_demo123]
Call CA_demo123 ended. Transcript:
  assistant: Hello! How can I help you today?
  user: Hey, can you check the weather for me
  assistant: Sure. What's your city or zip code?
  user: Los Angeles
  assistant: Okay, let me check that. It's sunny and 72 degrees in Los Angeles.

If you don't hear a greeting, double check that welcome_instruction is set.

Going further: placing outbound calls

Now that you have an agent callers can reach, let's build one that reaches out to them instead.

Place an outbound call

The same GPTLiveProvider that answers inbound calls can also place them — useful for proactive outreach like appointment reminders, order updates, or callback flows where the agent initiates contact instead of waiting for the caller to dial in. This continues in the same file and reuses the voice_channel you already built in Steps 1-4 , no new project or separate agent to set up.

Making outbound calls requires you comply with the various rules and regulations in your jurisdiction. For example, in the United States, your outbound calls have to comply with the Telephone Consumer Protection Act (or TCPA). We ask that you seek your own counsel when determining whether your usage is compliant. Your app also has to comply with Twilio’s Terms of Service and Voice Services Policies. 
from tac.channels.voice.media_streams.gpt_live import InitiateVoiceConversationOptionsGPTLive
await voice_channel.initiate_outbound_conversation(
    InitiateVoiceConversationOptionsGPTLive(to="+15551234567")
)

to must be in E.164 format (e.g., +15551234567), which is what Twilio's Voice API requires.

In the full example, this runs behind a --to flag. Start the server with python openai_gpt_live.py --to +15551234567 and it places the call as soon as the server starts, while still answering inbound calls as usual.

That's the entire outbound flow. TAC handles placing the call, connecting the Media Stream, and bridging it to GPT-Live the same way it does for an inbound call. From here, initiate_outbound_conversation is the one API you need whether you're calling one number or looping over a list to run a whole outbound campaign.

Other features to explore

This tutorial covers the basics, but Twilio Agent Connect and GPT-Live support more than what's shown here:

  • One SDK, every channel. TAC also ships providers for SMS, RCS, WhatsApp, and Chat, if your agent needs to talk to customers outside of voice.
  • Swappable voice backends. The same VoiceChannel API works with Conversation Relay and the OpenAI Realtime API.
  • Production-ready call handling. Status, answering-machine detection, and recording callbacks, plus programmatic hang-up, for turning a demo into something that survives real phone traffic.
  • Per-call session customization. Override the session config for a specific call without touching your channel-wide defaults.

Conclusion

You now have a phone number that connects callers to a GPT-Live-1-powered voice agent that greets them proactively and answers questions by calling a tool, and you can also build that same agent placing outbound calls on its own.

The same pattern works for any scenario where a caller needs a tool-using voice agent, from appointment booking to order status lookups, and extends naturally to per-caller prompts and other features when you need them.

From here, check out the Twilio Agent Connect API reference for the full VoiceProvider interface, or the OpenAI Realtime API provider if you need an alternative speech-to-speech backend.

Additional Resources

Xinghao Huang is a Software Engineer at Twilio. Off the clock, he cooks his way through both Chinese and Western cuisines, keeps a spice cabinet of a few dozen jars — all matching, naturally — better organized than most of his codebases, and maintains a personal recipe website. He can be reached at jahuang [at] twilio.com.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.