How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript

How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript

"It works great in my testing script, but the second we put it on a real phone call, it falls apart." That's a common complaint once a voice AI demo meets production traffic.

OpenAI's GPT-Live is a family of full-duplex speech-to-speech, or S2S, models. They separate the conversation and any necessary background work, allowing you to delegate to your own backend or an OpenAI hosted model.

Twilio Agent Connect is Twilio's open-source TypeScript SDK for building voice and messaging AI agents, and we now ship a GPTLiveProvider that bridges Twilio Programmable Voice Media Streams directly to GPT-Live-1 in the OpenAI API. It all works through a flexible, declarative config interface – you don’t need any custom WebSocket plumbing.

In this tutorial, you'll use TypeScript and build a phone number that connects inbound callers to a GPT-Live-powered voice agent. It’ll greet the caller and the caller can ask it a question that triggers a tool call. Then, I’ll show you how to place an outbound call with Twilio and GPT-Live.

While we build together, I’ll also show you other features like per-call customization, and you can explore from there. Let’s get started!

Prerequisites and common pitfalls to getting started

• A free Twilio account - Sign up for an account here.
• A Twilio phone number with Voice capability - buy one from your Twilio Console.
• An OpenAI API key from a project with GPT-Live access.
• Node.js 22.13+ and the Twilio Agent Connect SDK v2.4.0+ installed, along with dotenv (npm install twilio-agent-connect dotenv).
• A public HTTPS/WSS endpoint for local development — ngrok works well.

Build the app

The full working example lives at getting_started/examples/gpt-live. Follow along below, or clone the repo and run that example directly.

Step 1: Set up TAC and the Fastify server

Every TAC voice app starts with a Twilio Agent Connect instance and a VoiceChannel. VoiceChannel is provider-agnostic: the same class hosts Conversation Relay, GPT Realtime, and GPT-Live, depending on which config you pass it.

TACConfig.fromEnv() reads its settings from environment variables.

Create a .env file in your project root with TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN (found on your Console dashboard), TWILIO_API_KEY and TWILIO_API_SECRET (from API keys & tokens), TWILIO_PHONE_NUMBER (from Phone Numbers > Manage > Active Numbers), TWILIO_VOICE_PUBLIC_DOMAIN (your ngrok domain from the prerequisites step), and OPENAI_API_KEY, and the dotenv/config import above picks them up automatically.

Step 2: Configure the GPT-Live session and greeting

defaultSessionConfig is sent as GPT-Live's session.start payload the moment the model connects. At minimum it needs the model, instructions (the system prompt), and audio.format set to TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE.

Setting welcomeInstruction makes the agent speak first instead of waiting for the caller, which is useful for any call where the caller doesn't know they've reached an AI agent yet.

Step 3: Wire up a tool call

GPT-Live doesn't call functions directly – instead, it delegates tool calls either through Client Delegation (where you need to build the delegation integration) or through a Responses API model. In this tutorial, we’ll use Responses delegation.

Any tool created with defineTool() can be exposed this way: register it in tools: [...] so Twilio Agent Connect can execute it, and separately list its schema under delegation.responses.tools so the model knows it exists.

Keeping tool execution outside the real-time voice loop means your tool logic can be as slow (or complex) as it needs to be. Consider all the things you might include in your logic: a database lookup, an API call, or a business rule.

The 3-second setTimeout in the code above makes the strength of this model visible: run the demo and the call keeps going while the tool runs. GPT-Live delegates the decision to call a tool to a Responses API model, then Twilio Agent Connect runs the actual TypeScript function and speaks back the result.

And hopefully, that shows how straightforward OpenAI’s Responses delegation is. Of course,when you build your own agent, swap getWeather for a function that hits your own backend. For example, you might look up an order status, check appointment availability, or pull account details. All the same wiring applies without changing anything else here.

Step 4: Wire up the server

TACServer mounts the TwiML endpoint at /twiml by default.

Step 5: Point your Twilio number at the webhook

With the server running behind a public tunnel (like ngrok), tell Twilio where to send incoming calls:

• In the Twilio Console, go to Phone Numbers > Manage > Active Numbers and select your number.
• Under Voice Configuration, set "A call comes in" to Webhook, and enter https://<your-domain>/twiml.
• Set the HTTP method to POST, then save.

Now, you're ready to call in! ☎️

And there you have it: you now have a phone number that greets callers, answers questions, and calls real functions, with no WebSocket code required.

Run, test, troubleshoot

Call your Twilio number… if everything is set up correctly, you should hear the agent's welcomeInstruction immediately, without saying anything first. After the greeting, ask "what's the weather in Los Angeles?" — the model should invoke get_weather via Responses delegation and read back the result in the same turn.

Have fun talking to your agent – and when you hang up, the transcript accumulated in ConversationSession.metadata.transcript prints to your console if you register an onConversationEnded callback that reads it off the session when the call ends. Here’s an example:

If you don't hear a greeting, double check that welcomeInstruction is set.

Going further: placing outbound calls

Now that you have an agent callers can reach, let's build one that reaches out to them instead.

Place an outbound call

The same GPTLiveProvider that answers inbound calls can also place them. An outbound agent is useful for proactive outreach like appointment reminders, order updates, or callback flows where the agent initiates contact instead of waiting for a caller to dial in.

To complete the outbound agent demo, continue in the same file and reuse the voiceChannel you already built in Steps 1-4.

to must be in E.164 format (e.g., +15551234567), which is what Twilio's Voice API requires.

In the full example, this runs behind a --to flag. Start the server with npm run dev -- --to +15551234567 and it places the call as soon as the server starts, while still answering inbound calls as usual.

That's the entire outbound flow. TAC handles placing the call, connecting the Media Stream, and bridging it to GPT-Live the same way it does for an inbound call. From here, initiateOutboundConversation is the one API you need whether you're calling one number or looping over a list to run a whole outbound campaign.

Other features to explore

This tutorial covers the basics, but Twilio Agent Connect and GPT-Live support more than what's shown here:

• One SDK, every channel. TAC also ships providers for SMS, RCS, WhatsApp, and Chat, if your agent needs to talk to customers outside of voice.

• Swappable voice backends. The same VoiceChannel API works with Conversation Relay and the OpenAI Realtime API.

• Production-ready call handling. Status, answering-machine detection, and recording callbacks, plus programmatic hang-up, for turning a demo into something that survives real phone traffic.

• Per-call session customization. Override the session config for a specific call without touching your channel-wide defaults.

Conclusion

Congratulations! You now have a phone number that connects callers to a GPT-Live-1-powered voice agent, whether by picking up when they place a call or by dialing outbound. You agent greets people proactively, and can answer more advanced questions by calling a tool.

The same pattern I laid out in this post works for any scenario where a caller needs a tool-using voice agent. So whatever you’re building – from appointment booking to order status lookups – this example extends to your use case, and any per-caller prompts and other features when you need them.

From here, check out the Twilio Agent Connect API reference for the full VoiceProvider interface, or the OpenAI Realtime API provider if you need an alternative speech-to-speech backend.

Additional Resources

• Twilio Agent Connect documentation

• Twilio Programmable Voice Media Streams

• OpenAI’s GPT-Live API Docs

• OpenAI’s GPT-Live-1 in the API Resources

• OpenAI's GPT-Live announcement

   

Wen Zhu is a Software Engineer at Twilio. Away from the keyboard, Wen heads for the trails whenever the weather cooperates, is usually planning the next trip before the current one ends, and dials in shots on a home espresso machine with the kind of patience normally saved for chasing down flaky tests. Wen can be reached on LinkedIn.

Xinghao Huang is a Software Engineer at Twilio. Off the clock, he cooks his way through both Chinese and Western cuisines, keeps a spice cabinet of a few dozen jars — all matching, naturally — better organized than most of his codebases, and maintains a personal recipe website. He can be reached at jahuang [at] twilio.com.