Contact sales Start for free
Contact sales Start for free

How to Connect Your Twilio Agent to External APIs

Time to read:

September 15, 2026
Written by
Amanda Lange
Twilion

How to connect your Twilio agent to external APIs

The world is starting to increasingly rely on voice-enabled AI agents to get work done. But an agent can only do so much. Voice AI agents by themselves are able to hold conversations, but what happens if the agent needs to do something like retrieve customer data, look at an inventory, or book an appointment for a user?

In order to make your AI agent truly helpful, you need for that AI agent to have access to real time information. External APIs can provide that information. When your agent works together with an API, your agent is empowered to get information your users really need, and take actions on the user's behalf like viewing inventory, calendars, menus, and more.

In this tutorial, you will use C# .NET 9 to build a voice agent using Twilio Conversation Relay. Your agent will use tool calling from an LLM-driven conversation to dynamically fetch live data from an external REST API. This tutorial uses a simple API with no additional authentication requirements to showcase the potential of the AI tool. When you have completed the tutorial, you should understand the pipeline to interact with an external API, and how you could employ this functionality in your own builds.

This tutorial is geared towards .NET developers, but you can find other languages below:

Prerequisites

To complete this tutorial you will need the following:

Building the application

Step 1 - Set up the ASP.NET Core project

Your first step is creating a new folder and a .NET project. Go into your terminal and type the following: 

dotnet new web -n TwilioAgentApi
cd TwilioAgentApi

Step 2 - Install dependencies

Install NuGet packages you will need for your project by typing the following into your terminal: 

dotnet add package Twilio
dotnet add package DotNetEnv
dotnet add package OpenAI

These nuget packages are necessary for your project setup: The Twilio package will allow your application to interface with Twilio's services. The dotnetenv package allows you to import your environment variables into your solution using a .env file. The OpenAIpackage will be used to connect your solution to OpenAI.

Step 3 - Configure environment variables

This tutorial is simple enough not to require much information from your Twilio account. But you will need somewhere to safely store your OpenAI API key. Create a file called .env in your project folder. Add to that file the following text: 

OPENAI_API_KEY=VAxxxxxxxxxxx

Your OpenAI API Key is generated from OpenAI's dashboard. You shouldn't need any other keys in this file. However, if you decide later to call an API that has additional authentication, that key can be stored here as well.

Step 4 - Build the base application

You will use the very simple API, Cat Facts, in this demo. Our application is going to make a simple API call to request a "Cat Fact" from our agent. This API requires no additional authentication and has simple output, which makes it very useful for a demonstration.

Your new C# project will have a file in it called Program.cs. Open this file in your IDE of choice, and adjust it to have the following code: 

using DotNetEnv;
using TwilioAgentApi;
Env.TraversePath().Load();
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton<OpenAiService>();
var app = builder.Build();
app.UseWebSockets();
app.MapPost("/voice", (HttpRequest req) =>
{
   var host = req.Host.Value;
   var twiml = $"""
       <?xml version="1.0" encoding="UTF-8"?>
       <Response>
           <Connect>
               <ConversationRelay url="wss://{host}/ws" welcomeGreeting="Hello! Ask me for a cat fact." />
           </Connect>
       </Response>
       """;
   return Results.Content(twiml, "application/xml");
});
app.Map("/ws", async (HttpContext ctx, OpenAiService openAi) =>
{
   if (!ctx.WebSockets.IsWebSocketRequest)
   {
       ctx.Response.StatusCode = StatusCodes.Status400BadRequest;
       return;
   }
   using var ws = await ctx.WebSockets.AcceptWebSocketAsync();
   await ConversationRelayHandler.HandleAsync(ws, openAi);
});
app.Run();

This code is making a connection to a websocket to enable your agent. You are using Conversation Relay to build the connection between OpenAI and your voice-capable Twilio number, creating a voice agent that can hold a natural sounding conversation. Notice that you have added a simple greeting for your agent using TwiML. This greeting line can be adjusted as needed to give the user an initial prompt for interaction.

Step 5 - Handle Conversation Relay

You will need some additional code to connect your websocket to Conversation Relay. Do this by creating a new C# file, called ConversationRelayHandler.cs, in your project folder. 

using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
namespace TwilioAgentApi;
public record ConversationMessage(string Role, string Content);
static class ConversationRelayHandler
{
   public static async Task HandleAsync(WebSocket ws, OpenAiService openAi)
   {
       string? callSid = null;
       var messages = new List<ConversationMessage>();
       var buffer = new byte[8192];
       try
       {
           while (ws.State == WebSocketState.Open)
           {
               using var ms = new MemoryStream();
               WebSocketReceiveResult result;
               do
               {
                   result = await ws.ReceiveAsync(buffer, CancellationToken.None);
                   if (result.MessageType == WebSocketMessageType.Close)
                   {
                       await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, null, CancellationToken.None);
                       Console.WriteLine($"[{callSid}] Call ended");
                       return;
                   }
                   ms.Write(buffer, 0, result.Count);
               } while (!result.EndOfMessage);
               var json = Encoding.UTF8.GetString(ms.ToArray());
               using var doc = JsonDocument.Parse(json);
               var root = doc.RootElement;
               var msgType = root.TryGetProperty("type", out var t) ? t.GetString() : null;
               switch (msgType)
               {
                   case "setup":
                       callSid = root.TryGetProperty("callSid", out var sid) ? sid.GetString() : null;
                       Console.WriteLine($"[{callSid}] Call connected");
                       break;
                   case "prompt":
                       if (!root.TryGetProperty("last", out var last) || !last.GetBoolean())
                           break;
                       var userText = root.TryGetProperty("voicePrompt", out var vp) ? vp.GetString() ?? "" : "";
                       Console.WriteLine($"[{callSid}] Caller: {userText}");
                       messages.Add(new ConversationMessage("user", userText));
                       var responseText = await openAi.StreamResponseAsync(ws, callSid, messages);
                       messages.Add(new ConversationMessage("assistant", responseText));
                       break;
                   case "interrupt":
                       var spoken = root.TryGetProperty("utteranceUntilInterrupt", out var u) ? u.GetString() ?? "" : "";
                       Console.WriteLine($"[{callSid}] Interrupted after: '{spoken}'");
                       if (messages.Count > 0 && messages[^1].Role == "assistant")
                           messages.RemoveAt(messages.Count - 1);
                       break;
                   case "error":
                       var desc = root.TryGetProperty("description", out var d) ? d.GetString() : "";
                       Console.WriteLine($"[{callSid}] Conversation Relay error: {desc}");
                       break;
               }
           }
       }
       catch (WebSocketException)
       {
           Console.WriteLine($"[{callSid}] Call ended");
       }
   }
   public static Task SendJsonAsync(WebSocket ws, object payload)
   {
       var text = JsonSerializer.Serialize(payload);
       var bytes = Encoding.UTF8.GetBytes(text);
       return ws.SendAsync(bytes, WebSocketMessageType.Text, true, CancellationToken.None);
   }
}

This code is communicating with your websocket, breaking your voice inquiries down into conversation messages to be processed by the AI. Changing your voice responses to text, it then streams that text in real time to the AI in order to get fast responses.

This is one important component, but you still need to make the connection to OpenAI. You will do that in the next step.

Step 6 - Connect to OpenAI

In this step, you will configure tool function schemas using the OpenAI C# SDK. For this, create a file called OpenAiService.cs.

Write the system prompt instructing the agent when to execute external API calls based on user voice prompts. You'll see the prompt inside the SystemPrompt constant in the code below. You can adjust this to your needs. In this prompt, you make sure that the AI realizes it's being used for voice interaction, by reminding it not to use any bullet points or emojis when it communicates. 

using System.Net.WebSockets;
using System.Text.Json;
using OpenAI;
using OpenAI.Chat;
namespace TwilioAgentApi;
sealed class OpenAiService
{
   const string Model = "gpt-4o-mini";
   const string SystemPrompt = """
       You are a cat fact generator. If your user asks you for a cat fact you will respond with a cat fact.
       Speak naturally as if talking on the phone. Use plain sentences only. Do not use lists or bullet points. Do not use any emojis.
       When you are asked for a cat fact you must call the get_cat_fact tool to retrieve one from the API.
       Do not just make up facts. If you do not know the answer, respond with "I don't know."
       """;
   static readonly HttpClient _http = new();
   static readonly ChatTool CatFactTool = ChatTool.CreateFunctionTool(
       functionName: "get_cat_fact",
       functionDescription: "Retrieves a random cat fact from the catfact.ninja API.",
       functionParameters: BinaryData.FromString("""
           {
             "type": "object",
             "properties": {},
             "required": []
           }
           """)
   );
   readonly ChatClient _client;
   public OpenAiService()
   {
       var apiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY")
           ?? throw new InvalidOperationException("OPENAI_API_KEY is not set.");
       _client = new OpenAIClient(apiKey).GetChatClient(Model);
   }
   public async Task<string> StreamResponseAsync(WebSocket ws, string? callSid, List<ConversationMessage> messages)
   {
       var openAiMessages = new List<OpenAI.Chat.ChatMessage>
       {
           new SystemChatMessage(SystemPrompt)
       };
       foreach (var m in messages)
           openAiMessages.Add(m.Role == "user"
               ? new UserChatMessage(m.Content)
               : new AssistantChatMessage(m.Content));
       var fullResponse = "";
       try
       {
           var options = new ChatCompletionOptions
           {
               MaxOutputTokenCount = 300,
               Tools = { CatFactTool },
           };
           var stream = _client.CompleteChatStreamingAsync(openAiMessages, options);
           ChatFinishReason? finishReason = null;
           var toolCallsAcc = new Dictionary<int, ToolCallAccumulator>();
           await foreach (var update in stream)
           {
               if (update.FinishReason.HasValue)
                   finishReason = update.FinishReason.Value;
               foreach (var part in update.ContentUpdate)
               {
                   if (!string.IsNullOrEmpty(part.Text))
                   {
                       fullResponse += part.Text;
                       await ConversationRelayHandler.SendJsonAsync(ws,
                           new { type = "text", token = part.Text, last = false });
                   }
               }
               foreach (var tc in update.ToolCallUpdates)
               {
                   if (!toolCallsAcc.TryGetValue(tc.Index, out var acc))
                   {
                       acc = new ToolCallAccumulator();
                       toolCallsAcc[tc.Index] = acc;
                   }
                   if (!string.IsNullOrEmpty(tc.ToolCallId)) acc.Id = tc.ToolCallId;
                   if (!string.IsNullOrEmpty(tc.FunctionName)) acc.Name = tc.FunctionName;
                   var argUpdate = tc.FunctionArgumentsUpdate;
                   if (argUpdate != null && !argUpdate.ToMemory().IsEmpty)
                       acc.Arguments += argUpdate.ToString();
               }
           }
           if (finishReason == ChatFinishReason.ToolCalls && toolCallsAcc.Count > 0)
           {
               var toolCalls = toolCallsAcc.Values.ToList();
               var assistantMsg = new AssistantChatMessage(
                   toolCalls.Select(tc => ChatToolCall.CreateFunctionToolCall(
                       tc.Id ?? "",
                       tc.Name ?? "",
                       BinaryData.FromString(string.IsNullOrEmpty(tc.Arguments) ? "{}" : tc.Arguments))).ToList()
               );
               openAiMessages.Add(assistantMsg);
               foreach (var tc in toolCalls)
               {
                   var result = await ExecuteToolAsync(tc.Name ?? "");
                   Console.WriteLine($"[{callSid}] Tool {tc.Name}({tc.Arguments}) -> {result}");
                   openAiMessages.Add(new ToolChatMessage(tc.Id ?? "", result));
               }
               fullResponse = "";
               var options2 = new ChatCompletionOptions { MaxOutputTokenCount = 400 };
               var stream2 = _client.CompleteChatStreamingAsync(openAiMessages, options2);
               await foreach (var update in stream2)
                   foreach (var part in update.ContentUpdate)
                       if (!string.IsNullOrEmpty(part.Text))
                       {
                           fullResponse += part.Text;
                           await ConversationRelayHandler.SendJsonAsync(ws,
                               new { type = "text", token = part.Text, last = false });
                       }
           }
       }
       finally
       {
           await ConversationRelayHandler.SendJsonAsync(ws,
               new { type = "text", token = "", last = true });
       }
       Console.WriteLine($"[{callSid}] Assistant: {fullResponse}");
       return fullResponse;
   }
   static async Task<string> ExecuteToolAsync(string name)
   {
       if (name != "get_cat_fact")
           return "Unknown tool.";
       try
       {
           var json = await _http.GetStringAsync("https://catfact.ninja/fact");
           using var doc = JsonDocument.Parse(json);
           return doc.RootElement.TryGetProperty("fact", out var f)
               ? f.GetString() ?? "No fact returned."
               : "No fact returned.";
       }
       catch (Exception ex)
       {
           return $"Error retrieving cat fact: {ex.Message}";
       }
   }
}
sealed class ToolCallAccumulator
{
   public string? Id { get; set; }
   public string? Name { get; set; }
   public string Arguments { get; set; } = "";
}

The ExecuteToolAsync task is what actually calls our external API. It reaches out to the API located at https://catfact.ninja and parses the json response from the API. If it can't find a fact, say, if the connection to the API is interrupted, it returns an error.

Testing your application

Now it is time to test your application and chat with your AI.

First, run your application using this command in the terminal: 

dotnet run

Once your webhook is running, you will need to expose it to the internet by using ngrok or another tunneling service. 

ngrok http localhost:5000

Replace 5000 with whatever port your application is running on if you have a different port.

Now ngrok will provide you with a url for utilizing in your Twilio console. Go into your Twilio console and find the Twilio phone number that you prepared. Under the option A Call Comes In, choose Webhook, and fill in your ngrok URL followed by /voice, as shown in the graphic below:

configuration for webhooks in the Twilio console
configuration for webhooks in the Twilio console

Be sure also that your HTTP block is set to POST.

Now save this configuration, and call your Twilio Phone Number.

You should hear a message with the AI greeting that you provided in Program.cs.

Try asking your AI about a cat fact and you will get a cat fact from the cat fact API!

Troubleshooting

If you are having some difficulty with your call, there are some common problems you might want to check. First of all, make sure your ngrok URL is correct in the console and matches the one that's in your terminal, with /voice appended to the end.

If you are still having issues, check your environment variables. You will need to make sure your API keys are correct for any key that you happen to be using, including your key for OpenAI. The sample API requires no additional keys, but if you decide to expand the application, you will also need to authenticate any external APIs that you call. Check the rules for your individual APIs.

Conclusion

Connecting LLM function tools to external HTTP endpoints empowers Twilio voice agents with real-time data. With the use of external APIs, you can create an agent that doesn't just respond to questions, but truly does the work your customers need.

Are you looking for some further project ideas or further reading? We also have a series on getting started creating your AI Phone Agent with Conversation Relay. Or check out how to do function and tool calling in node.JS or your language of choice!

If you got stuck at any point during this tutorial, the full solution is available for reference on Github.

We can't wait to see what we can help you build!

Amanda Lange is a .NET Engineer of Technical Content. She is here to teach how to create great things using C# and .NET programming. She can be reached at amlange [ at] twilio.com.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.