Build an AI Voice Assistant with Twilio Voice and Media Streams, OpenAI's GPT-Live API, and Node.js

Time to read:

September 10, 2026

Written by

Paul Kamp

Twilion

Reviewed by

Lenore Files

Twilion

Ryan Rouleau

Twilion

Build an AI Voice Assistant with Twilio Voice and Media Streams, OpenAI's GPT-Live API, and Node.js

We're so excited that our friends at OpenAI have launched GPT-Live-1 in the OpenAI API! GPT-Live-1 is the first model in their new GPT-Live family – the speech-to-speech models behind the new ChatGPT Voice.

GPT-Live changes how you can build agents with speech-to-speech models. It runs a conversation across two models:

a speech-to-speech model owns the audio - listening, talking, and handling interruptions
a reasoning model does the thinking - calling tools where needed

OpenAI calls the handoff process delegation. Practically, your caller talks with a responsive voice model while another model stands by in the background for when you need deeper reasoning, web search, or more complex work done.

In this tutorial, I'll show you how to build an AI voice assistant powered by OpenAI’s GPT-Live-1 in their API. You can call it from your phone, carried by Twilio Programmable Voice and Twilio Media Streams, with a server written in Node.js. The assistant will answer phone calls, greet callers, and carry on a thrilling, avian-and-rock-legend-tinged conversation. The agent can also search the web, and make tool calls to functions you define.

Enough preening – let's fly! 🦉

Prerequisites and common pitfalls

To follow along with this tutorial, you'll need:

A Twilio account – sign up for a free account if you don't yet have one.
A Twilio phone number with Voice capability – here's how to search and buy a phone number.
An OpenAI account with GPT-Live-1 access:
- An API key with permission to use the gpt-live-1 and gpt-5.6-terra models.
Node.js 22 or later – I used 24.5.0 writing this tutorial.
A way to expose localhost to the internet – ngrok is the usual choice.
A phone. Seriously, any phone… you're going to call your agent pretty soon!

Let's do this.

In a hurry? The finished application is on GitHub if you'd rather skip ahead.

Build the app

Step 1: Initialize the GPT-Live agent project

Fire up your console – it's time to start a project.

Make a directory and initialize it (so npm has a package.json where it can write dependencies):

mkdir live-demo
cd live-demo
npm init -y

Step 2: Install dependencies

We need a web server that speaks HTTP and WebSockets. That's a perfect job for Fastify, which handles the server side (along with ws handling the client side). We'll also use dotenv to keep your credentials out of the code.

npm install fastify @fastify/formbody @fastify/websocket ws dotenv

For reference, here's what I had installed:

+-- @fastify/formbody@8.0.1
+-- @fastify/websocket@11.0.1
+-- dotenv@16.4.5
+-- fastify@5.0.0
`-- ws@8.18.0

Step 3: Create the project files

Now, you need two files – one for secrets, one for the server code. And we're using ES modules in this tutorial, so go ahead and add "type": "module" to your package.json while you're in there anyway:

touch .env live-demo.js

Put your OpenAI API key in .env:

OPENAI_API_KEY=<your-openai-api-key>
PORT=5050

Add the module type to package.json:

{
  "type": "module"
}

Step 4: Set up your imports and configuration

Like any other Node project, we'll start with some imports.

Open live-demo.js and paste this:

import Fastify from 'fastify';
import WebSocket from 'ws';
import dotenv from 'dotenv';
import fastifyFormBody from '@fastify/formbody';
import fastifyWs from '@fastify/websocket';
dotenv.config();
const { OPENAI_API_KEY } = process.env;
if (!OPENAI_API_KEY) {
   console.error('Missing OpenAI API key. Please set it in the .env file.');
   process.exit(1);
}
const MODEL = 'gpt-live-1';
const DELEGATED_MODEL = 'gpt-5.6-terra';
const VOICE = 'marin';
const USER_AGENT = 'twilio-demos/Node 1.0.0';
const PORT = process.env.PORT || 5050;

Here's how we're going to use these:

MODEL is the speech-to-speech model that talks to your caller.
DELEGATED_MODEL is the reasoning model it hands hard questions to.
VOICE is the voice your caller actually hears. I used marin – see OpenAI’s docs for the rest.
USER_AGENT is how your app introduces itself to OpenAI, in the form [company or library]/[language] [version]. Put your own name in this one, though I did supply an example…
PORT is the port your server listens on, read from .env with a fallback to 5050.

My take with this pattern? It’s convenient! You can swap either model or the voice without touching another line, which is handy when you're tuning the agent’s performance later.

Step 5: Write your prompts

Because we'll be dealing with a couple of models, we will set different instructions based upon which job they'll be doing. Paste these next:

const OPENING = "Hi there! I am an AI voice assistant powered by Twilio and OpenAI's GPT-Live. How can I help?";
const VOICE_PROMPT = "You are an AI voice assistant powered by Twilio and OpenAI's GPT-Live. "
    + 'Never call yourself ChatGPT. '
    + 'You are helpful and bubbly, love to chat about anything the caller is interested in, and are prepared to offer facts. '
    + 'You have a penchant for dad jokes, owl jokes, and rickrolling – subtly. '
    + 'Always stay positive, but work in a joke when appropriate. '
    + 'You can look up the top headline for any city, and you can search the web for real facts. '
    + 'If the caller asks what you can do, tell them those two things. '
    + 'Name the city you are looking up so the lookup is unambiguous.';
const BACKEND_PROMPT = 'Use get_top_headline for the local headline of the day and web_search for real facts. '
    + 'Answer in one or two sentences.';

VOICE_PROMPT shapes your agent's identity and personality – it controls how your assistant sounds, what it says it can do, and (sometimes) which jokes it'll tell. Notice we lead the prompt with the AI disclosure – if a caller asks later who they're talking to, this is what the model draws on.

BACKEND_PROMPT goes to the delegation model and defines behavior when that model is needed – here, we're instructing it on which tool to reach for and how long an answer to give. And asking for a sentence or two? Since this conversation is by voice, we're keeping it tight!

And, you guessed it, OPENING is your assistant's first spoken line, verbatim.

Step 6: Define the tools your assistant can call

With the instructions out of the way, next up, we're going to set up our tool calls.

const TOOLS = [
    { type: 'web_search' },
    {
        type: 'function',
        name: 'get_top_headline',
        description: "Get today's top headline for a city.",
        parameters: {
            type: 'object',
            properties: { city: { type: 'string' } },
            required: ['city'],
            additionalProperties: false
        }
    }
];

{ type: 'web_search' } is the configuration for hosted web search – we're using OpenAI's internal tool here.

get_top_headline, on the other hand? That's yours – treat it as a template or a pattern, though, since I assume you don't want your production agent to make city-simulator game inspired jokes. Whatever tool calls you make should house your own business logic.

With get_top_headline, the description and parameters dictate how the reasoning model decides what to call – and if it picks your tool, what to pass. You should write them as you would documentation for a colleague who can't see your code.

Now, I'll show you the (currently silly) function itself, plus some headlines for it to return. Paste these lines next:

const HEADLINES = [
    'Fusion plant comes online; residents petition to have it moved next to the airport',
    'Traffic permanently solved by new roundabout. 400 citizens now on their third hour circling it',
    'Mayor renames every street "Main Street"',
    'Godzilla attack downgraded to "moderate inconvenience" by tourism board',
    'City votes to replace bus network with a single party bike',
    'New arcology opens to complaints the clouds are too close',
    'Water treatment plant water rated "mostly delicious" by local food critic',
];
// Mock tool call with fake data, slow on purpose.
const getTopHeadline = async ({ city }) => {
    console.log('Reticulating splines...');
    await new Promise((resolve) => setTimeout(resolve, 2000));
    return { city, headline: HEADLINES[Math.floor(Math.random() * HEADLINES.length)] };
};

Reticulating splines, indeed!

The function is pretty boilerplate, but for this demo I'll note the two-second delay is deliberate. Your real tools might hit a database or a slow API, so a demo where everything returns instantly isn't a real-life demo. (City simulator 🤝 Agent simulator.)

Step 7: Answer the call with TwiML

Okay, and now we're on to the part where your agent leaves the nest and picks up the phone.

When someone dials your Twilio number, Twilio doesn't know what to do unless you tell it – that's where you answer with TwiML, Twilio's Markup Language. The verb we want is <Connect><Stream> – it opens a bidirectional WebSocket for the life of the call.

const fastify = Fastify();
fastify.register(fastifyFormBody);
fastify.register(fastifyWs);
fastify.get('/', async () => ({ message: 'Twilio Media Stream Server is running!' }));
fastify.all('/incoming-call', async (request, reply) => {
    const host = request.headers['x-forwarded-host'] || request.headers.host;
    reply.type('text/xml').send(`<?xml version="1.0" encoding="UTF-8"?>
<Response><Connect><Stream url="wss://${host}/media-stream" /></Connect></Response>`);
});

The / route is a heartbeat, so you can confirm your tunnel works in your browser before you spend a phone call on it. (Try opening it.)

/incoming-call is the webhook you'll point your Twilio number at in Step 10. Twilio requests it the moment a call arrives, and whatever TwiML you return is what happens on that call – here, <Connect><Stream> hands the audio over to the WebSocket route you'll write next. It's registered with fastify.all so it answers whether Twilio sends a GET or a POST.

The x-forwarded-host check on the first line deserves a moment, too. You have to tell Twilio which wss:// URL to dial, and you can build it from the request's Host header – or, in the case of a forwarded tunnel (like you are probably using for this tutorial) x-forwarded-host. We read x-forwarded-host first, then fall back to Host.

Step 8: Open the GPT-Live session

Now it's time for the more interesting part… you'll register the WebSocket route, then reach out to OpenAI. Paste this next:

fastify.register(async (fastify) => {
   fastify.get('/media-stream', { websocket: true }, (connection) => {
       console.log('Client connected');
       let streamSid = null;
       let sessionRequested = false;
       let sessionReady = false;
       const openAiWs = new WebSocket('wss://api.openai.com/v1/live/sessions', {
           headers: {
               Authorization: `Bearer ${OPENAI_API_KEY}`,
               'User-Agent': USER_AGENT
           }
       });
       const send = (event) => {
           if (openAiWs.readyState === WebSocket.OPEN) openAiWs.send(JSON.stringify(event));
       };
       const close = () => {
           sessionReady = false;
           connection.close();
           openAiWs.close();
       };

The second header there is a request from OpenAI: identify your app with a User-Agent so they can tell your traffic apart from everyone else's. Mine says Twilio because this is a Twilio demo – yours should reflect your app.

Next, configure the session. This is the message that sets up everything we defined in Steps 4 through 6 – that is, the personality, voice, and audio format, plus details for how delegation should work:

// Wait for Twilio's stream ID before starting the session.
       const startSession = () => {
           if (sessionRequested || !streamSid || openAiWs.readyState !== WebSocket.OPEN) return;
           sessionRequested = true;
           send({ type: 'session.start', session: {
               model: MODEL,
               instructions: VOICE_PROMPT,
               audio: { format: { type: 'audio/pcmu', rate: 8000 }, output: { voice: VOICE } },
               delegation: {
                   type: 'responses',
                   responses: { model: DELEGATED_MODEL, instructions: BACKEND_PROMPT, tools: TOOLS }
               }
           } });
       };
       openAiWs.on('open', () => {
           console.log('Connected to GPT-Live-1');
           startSession();
       });

Got a second to talk? Let's discuss this code a little more.

startSession guards the handshake against races. It checks sessionRequested, and ensures the OpenAI socket is open and streamSid (Twilio’s call ID) is present. This prevents race conditions, and ensures your session is configured once.

session.start is the handshake. It carries session.model and the session configuration, and GPT-Live-1 replies with session.started when ready. That reply is where you make the assistant talk first – and where you make it say your OPENING verbatim. I'll show that in Step 9.

delegation is OpenAI’s two-model setup. type: 'responses' routes turns to the Responses API when needed. Everything nested under responses (model, instructions, and tools) belongs to the reasoning, or delegation, model.

Step 9: Relay events between Twilio and OpenAI

Now we come to the event logic.

openAiWs.on('message', async (data) => {
           try {
               const event = JSON.parse(data);
               if (event.type === 'session.started') {
                   sessionReady = true;
                   // Quote this ID if you ever need OpenAI's help with a call.
                   console.log('GPT-Live-1 session', event.session?.id);
                   send({ type: 'session.instructions.append', delegation_id: null, content: `Your first spoken line on this call is, verbatim: "${OPENING}"` });
                   send({ type: 'session.commentary.append', delegation_id: null, content: OPENING });
               } else if (event.type === 'session.output_audio.delta' && streamSid && connection.readyState === WebSocket.OPEN) {
                   connection.send(JSON.stringify({ event: 'media', streamSid, media: { payload: event.delta } }));
               } else if (event.type === 'response.event'
                   && event.event?.type === 'response.output_item.done'
                   && event.event.item?.type === 'function_call' && event.event.item.status === 'completed') {
                   const { call_id, name, arguments: args } = event.event.item;
                   console.log('Tool call:', name, args);
                   const output = name === 'get_top_headline'
                       ? await getTopHeadline(JSON.parse(args))
                       : { error: 'unknown tool' };
                   send({ type: 'response.item.create',
                       item: { type: 'function_call_output', call_id, output: JSON.stringify(output) } });
                   send({ type: 'response.create' });
               } else if (event.type === 'session.output_transcript.delta') {
                   console.log('Assistant:', event.delta);
               } else if (event.type === 'error') {
                   console.error('GPT-Live-1 error:', event.error);
               }
           } catch (error) { console.error('Error processing the GPT-Live-1 message:', error); }
       });

Five branches, one bird 🦉. Let's discuss them:

session.started is GPT-Live-1 confirming it's ready – and it carries the session id, which OpenAI needs if you ever ask them about a call. Right after that ID lands, we send two events: session.instructions.append with a "say this verbatim" directive, and session.commentary.append to trigger the model into speaking. Together they make GPT-Live-1 open the call with your exact OPENING – so the AI disclosure lands word-for-word (handy for compliance), and the assistant speaks first instead of waiting for the caller.
session.output_audio.delta carries your assistant's voice – the base64 μ-law bytes are in event.delta, and we hand them to Twilio unchanged.
The function_call branch is where delegation calls come back. Function events arrive inside a response.event envelope – dispatch on the inner event.type and read the completed function item's call_id, name, and JSON arguments. Run your function, then answer with two events: response.item.create to append the function_call_output, and response.create to continue the response.
session.output_transcript.delta is the text of whatever your assistant just said, in 200 ms increments – handy for logs and captions.
error logs and keeps going – deliberately.

Now, the Twilio side:

connection.on('message', (message) => {
           try {
               const data = JSON.parse(message);
               if (data.event === 'media' && sessionReady && openAiWs.readyState === WebSocket.OPEN) {
                   send({ type: 'session.input_audio.append', audio: data.media.payload });
               } else if (data.event === 'start') {
                   streamSid = data.start.streamSid;
                   console.log('Incoming stream has started', streamSid);
                   startSession();
               } else if (data.event === 'stop') {
                   close();
               }
           } catch (error) { console.error('Error parsing Twilio message:', error); }
       });

Twilio's start event is the other half of the handshake in Step 8. Here we grab the streamSid and try to start the session again.

media frames are the caller's voice (gated on sessionReady so you're not shouting at a session that hasn't opened yet), while stop means the call ended.

And finally, the cleanup handlers (and the server start). Paste this next:

connection.on('close', () => { close(); console.log('Client disconnected.'); });
       connection.on('error', close);
       openAiWs.on('close', (code, reason) => {
           close();
           console.log('Disconnected from GPT-Live-1', code, reason.toString());
       });
       openAiWs.on('error', (error) => { console.error('Error in the OpenAI WebSocket:', error); close(); });
   });
});
fastify.listen({ port: PORT }, (err) => {
   if (err) { console.error(err); process.exit(1); }
   console.log(`Server is listening on port ${PORT}`);
});

And that's the entire application! On to the run...

Step 10: Expose your server and configure Twilio

Go ahead and start your server:

node live-demo.js

You should see:

Server is listening on port 5050

Now expose port 5050. With ngrok, you would run:

ngrok http 5050

Copy the forwarding hostname (without the scheme) – it'll be something like abc123.ngrok.app. Open it in a browser and confirm:

{"message":"Twilio Media Stream Server is running!"}

If you see that, your tunnel is public and your server is alive! Of course, if you get a login page or a warning screen instead, fix that before you go any further – Twilio can't click through.

Next, we will point your number at it. In the Twilio Console, go to Phone Numbers → Manage → Active numbers, click your voice-capable number. Then, under Voice Configuration, set A call comes in to Webhook with:

https://abc123.ngrok.app/incoming-call

(Swapping in your server from above.) Leave the method as HTTP POST and Save.

Run and test your assistant

We're almost there, can you feel it? Well, even if you can't... you're about to hear it.

Dial your Twilio number. Your agent should pick up and greet you. Wait for the agent to finish, then try a few scenarios:

Ask what your agent can do. It's nice to know if your agent knows its own powers!

Ask for a headline. "What's the top news headline for Boston, Massachusetts?" This is how to test your tool call – though, of course, you can pick a city other than Boston (but why would you want to?).

Tool call: get_top_headline {"city":"Boston, Massachusetts"}
Reticulating splines...
Assistant:  Today's top headline in Boston is: Mayor renames every street "Main Street."
Assistant:  That could make giving directions a little... mainstream.

Ask for a fact. "Who wrote A Tale of Two Cities, and what year was it published?" Again, this should flex a tool call, except this time OpenAI's hosted web_search tool (you won't see delegation instructions in the terminal).

Interrupt it. I like to ask the agent to count to twenty, then cut it off. If you ask where it counted to, it should be reasonably close (within a few numbers).

Isn't that a hoot?

Troubleshooting

The call connects but I hear nothing. Check that your TwiML contains a public hostname: curl -X POST https://your-tunnel/incoming-call and read the wss:// URL.

Missing OpenAI API key. Your .env isn't being read, or you mispasted your key. Confirm the .env file sits in the same directory as live-demo.js and you have a working OpenAI API Key in there (with access to GPT-Live and the delegation model, which is gpt-5.6-terra if you haven't tweaked it).

Something looks wrong on OpenAI's side. Copy the session ID your terminal printed when the call started and include it when you ask OpenAI for help. Sessions that arrive through a provider are hard to locate without it.

Conclusion

Congratulations! You built an AI voice assistant on GPT-Live-1 that answers a phone call, greets you, and carries on a conversation. It also searches the web, calls your own code, and has a predilection for 80s British rockers. And all that in under 200 lines!

And now, the code is ready for your modifications: swap in your business logic, rewrite the prompts, and enjoy GPT-Live. And I bet you will…

Additional resources

Paul Kamp is the Technical Editor-in-Chief at Twilio. He still regrets cutting the Transit Authority budget in SimCity 2000 (yes, he was warned). He can be reached at pkamp [at] twilio.com.

Related Resources

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.

Build an AI Voice Assistant with Twilio Voice and Media Streams, OpenAI's GPT-Live API, and Node.js

Related Posts

Related Resources