Add Function Calling to Your Twilio Voice and Claude ConversationRelay Integration

Time to read: 5 minutes

July 03, 2025
Written by
Paul Kamp
Twilion
Reviewed by
Amanda Lange
Twilion
Kelley Robinson
Twilion
Kat McCormick Sweeney
Twilion
Brandon Hawkins
Twilion

Add Function Calling to Your Twilio Voice and Claude ConversationRelay Integration

 

ConversationRelay lets you build real-time, interactive, human-like voice experiences by connecting Twilio Voice to any AI Large Language Model via a WebSocket.

We previously explored connecting Twilio Voice to Anthropic’s AI Model Claude, then building streaming and interruption-aware conversation tracking into our implementation. But what if you want your AI to reach beyond its own knowledge – to call APIs dynamically when triggered by a caller’s request, or use your own internal tools… like a digital assistant?

That’s where Tool Use or Function Calling comes in.

Let’s walk step-by-step through adding Tool Calling to your Twilio and Claude AI agent application. In this build, you’ll enable your assistant to fetch programming jokes from an external service at a caller’s request… and lay the foundation for all kinds of LLM-agent workflows that you’ll want to build next.

Let’s get started!

Prerequisites

You’ll need:

  • Node.js installed (I tested with 23.9.0, but other versions should work)
  • A Twilio Account and phone number with voice support
  • An Anthropic account with API key access
  • ngrok or another tunneling tool
  • An IDE, text editor, and your phone (for testing voice)
Not in the mood to follow step-by-step? You can clone the working app ( sans API Keys and ngrok URL) from this branch. This tutorial builds on top of step3-conversation-tracking in our repo, but you can also build from scratch. You can also revisit the previous tutorial if you'd like a refresher.

Get the example repo

For a smooth experience, clone the workshop repo. Branches label each major step:

git clone https://github.com/pkamp3/cr-anthropic-demo.git
cd cr-anthropic-demo
git checkout step4-tool-calling

Or, follow along and update your own project based on my code and explanations below.

Set up your environment: .env and install dependencies

Whether updating an existing project or starting new, make sure you have a .env file with:

ANTHROPIC_API_KEY="sk-ant-..."
NGROK_URL="abcdefgh.ngrok.app"

These placeholder values should be replaced with a working API key, and your ngrok URL (which you will generate below).

Be careful to never commit your API keys to your repo. Add .env to your .gitignore if you’re working in git.

Install dependencies: 

npm install fastify @fastify/websocket @fastify/formbody @anthropic-ai/sdk dotenv axios
npm pkg set type="module"

How tool calling works in Claude

Tool calling or tool use (also called function calling) allows your assistant to choose, based on user input and your initial system prompt, to call an external function or API endpoint you describe. Claude models expose this through a tools array when generating messages, and returns special tool_use content blocks as part of its streaming response.

We’ll also:

  • Describe our available "tool" (fetching a programming joke) in our prompt and in our tools array.
  • Let Claude decide when to invoke our tool, based on the prompt and our instructions
  • On a tool call, fetch data (a programming joke) and add it to the conversation, inform the LLM that ‘it’ told a joke, then resume the conversation as normal

1. Update prompts and tool definitions in server.js

First, let’s tweak our greetings and system prompt.

These clarify (for you, the developer, and the model) what capability you’re giving Claude. Pay close attention to what you are adding about get_programming_joke:

const WELCOME_GREETING = "Hi! I am a voice assistant powered by Twilio and Anthropic. Ask me anything!";
const SYSTEM_PROMPT = `You are a helpful assistant. This conversation is being translated to voice, so answer carefully. 
When you respond, please spell out all numbers, for example, twenty not 20. Do not include emojis in your responses. Do not include bullet points, asterisks, or special symbols.
You should use the 'get_programming_joke' function only when the user is asking for a programming joke (or a very close prompt, such as developer or software engineering joke). 
For other requests, including other types of jokes, you should use your own knowledge.`;

This prompt tells Claude when to reach out to the external joke API and when to stick to its built-in knowledge.

2. Add the tool fetching function

Let’s define the function our agent can "call": the ability to fetch programming jokes from jokeapi.dev.

Add this near the top of your server.js:

import axios from "axios";
async function getJoke() {
  const response = await axios.get("https://v2.jokeapi.dev/joke/Programming?safe-mode");
  const data = response.data;
  return data.type === "single"
    ? data.joke
    : `${data.setup} ... ${data.delivery}`;
}

3. Expand aiResponseStream to handle tool calls

This is the heart of the feature, so let’s break it down.

In the previous tutorial, you streamed tokens from Claude. Now, you both stream tokens and look for special tool_use blocks. When you see one, you:

  • Fetch the needed info from your tool (e.g., get a joke)
  • "Push" both the LLM trigger (tool use) and your tool result into local conversation state,
  • Then send the joke (or other tool output) immediately down the WebSocket to Twilio for ConversationRelay to handle the text-to-speech step.

Your updated functions (focus on if (chunk.type === 'content_block_start' && chunk.content_block.type === 'tool_use') {):

 

async function aiResponseStream(conversation, ws) {
  const tools = [
    {
      name: "get_programming_joke",
      description: "Fetches a programming joke",
      input_schema: {
        type: "object",
        properties: {},
        required: []
      }
    }
  ];
  const stream = await anthropic.messages.create({
    model: "claude-sonnet-4-20250514", // Update to your Claude version
    max_tokens: 1024,
    messages: conversation,
    system: SYSTEM_PROMPT,
    tools: tools,
    stream: true,
  });
  const assistantSegments = [];
  console.log("Received response chunks:");
  for await (const chunk of stream) {
    if (chunk.type === 'content_block_delta' && chunk.delta.type === 'text_delta') {
      const content = chunk.delta.text;
      ws.send(JSON.stringify({
        type: "text",
        token: content,
        last: false,
      }));
      assistantSegments.push(content);
    }
    // This block fires when Claude triggers *your tool*
    if (chunk.type === 'content_block_start' && chunk.content_block.type === 'tool_use') {
      const toolCall = chunk.content_block;
      if (toolCall.name === "get_programming_joke") {
        const joke = await getJoke();
        // Add tool use and tool result to conversation
        conversation.push({
          role: "assistant",
          content: [{
            type: "tool_use",
            id: toolCall.id,
            name: toolCall.name,
            input: toolCall.input
          }]
        });
        conversation.push({
          role: "user",
          content: [{
            type: "tool_result",
            tool_use_id: toolCall.id,
            content: joke
          }]
        });
        ws.send(JSON.stringify({ type: "text", token: joke, last: true }));
        assistantSegments.push(joke);
        console.log("Fetched joke:", joke);
      }
    }
  }
  ws.send(JSON.stringify({ type: "text", token: "", last: true }));
  console.log("Assistant response complete.");
  // Save completed assistant response to session (for interruption, context, etc.)
  const sessionData = sessions.get(ws.callSid);
  sessionData.conversation.push({ role: "assistant", content: assistantSegments.join("") });
}

What’s going on here?

  • We define the tool metadata (in the const tools array I discussed above).
  • When you are watching the streaming tokens coming back from Claude, your logic will watch for content_block_start with type: 'tool_use'. If you see that, Claude is asking you to call your tool!
  • It’s straightforward from there: you fetch the result using your own logic, push the tool use and result into your conversation (so Claude can see the results, or what ‘it’ said, on its turn), and send the joke’s text to ConversationRelay.

Why is there so much bookkeeping in conversation? We want Claude to have a full, accurate “memory” of the conversation (something you might need to watch for non-idempotent tool uses!) including tool requests and responses on each conversation turn.

4. Update WebSocket conversation handling

This part stays mostly the same as in our previous step, except now our session and call storage uses a slightly different data structure for tracking the ongoing conversation.

Look for this case in your WebSocket handling logic:

case "setup":
  const callSid = message.callSid;
  ws.callSid = callSid;
  sessions.set(callSid, { conversation: [] });
  break;
case "prompt":
  const sessionData = sessions.get(ws.callSid);
  sessionData.conversation.push({ role: "user", content: message.voicePrompt });
  aiResponseStream(sessionData.conversation, ws);
  break;

That’s it! Everything else downstream (including streaming, forwarding tokens, and recognizing interruptions) continues to Just Work™️. More on that in a second.

5. Interruption handling

If you followed my prior tutorial, you already track interruptions by truncating the last assistant message and removing any "future" turns past the interruption point. That logic is unchanged:

function handleInterrupt(callSid, utteranceUntilInterrupt) {
  const sessionData = sessions.get(callSid);
  const conversation = sessionData.conversation;
  let updatedConversation = [...conversation];
  const interruptedIndex = updatedConversation.findLastIndex(
    (message) =>
      message.role === "assistant" &&
      message.content.includes(utteranceUntilInterrupt)
  );
  if (interruptedIndex !== -1) {
    const interruptedMessage = updatedConversation[interruptedIndex];
    const interruptPosition = interruptedMessage.content.indexOf(utteranceUntilInterrupt);
    const truncatedContent = interruptedMessage.content.substring(0, interruptPosition + utteranceUntilInterrupt.length);
    updatedConversation[interruptedIndex] = {
      ...interruptedMessage,
      content: truncatedContent,
    };
    updatedConversation = updatedConversation.filter(
      (message, index) =>
        !(index > interruptedIndex && message.role === "assistant"),
    );
  }
  sessionData.conversation = updatedConversation;
  sessions.set(callSid, sessionData);
}

And in your WebSocket handler: 

case "interrupt":
  handleInterrupt(ws.callSid, message.utteranceUntilInterrupt);
  break;
Depending on what you are doing with your tool calls, you may want to look at ConversationRelay attributes preemptible and interruptible to control the agent’s behavior while you call tools.

6. Test your AI agent with live tool calling

Ready to go? Fire it up and give it a call!

1. Run ngrok:

ngrok http 8080

2. Set your NGROK_URL in .env, omitting the scheme (‘http://’ or ‘https://’) – e.g., NGROK_URL="abcdefgh.ngrok.app":

Screenshot showing an active Ngrok tunnel with session status online and a red arrow pointing at the forwarding URL.

3. Start your server: 

node server.js

4. In your Twilio Console, go to Phone Numbers > Settings > Active Numbers. Under A call comes in in Voice Configuration, select Webhook, HTTP POST, and set the URL to your ngrok URL followed by the path /twiml:

A screenshot showing Twilio console call service

5. Save

Awesome!

Now, call your Twilio number. You’ll hear the greeting, then have the opportunity to chat with Claude. Ask for a programming joke to trigger the tool (and watch your console).

Try: “Can you tell me a programming joke?”

Claude will ask you to invoke the tool, call the Joke API in clean mode, and you’ll hear the joke streamed back as soon as it’s fetched! I hope you laugh…

What else can you build with Claude and Twilio ConversationRelay?

By adding tool calling to your integration, you just gave your Twilio and Claude voice AI agent agency: it can reach inside your company or outside on the internet, use tools, and make your voice application richer and more interactive. Try connecting to different APIs, or branch further and experiment with complex tool schemas, workflows, or even full assistants.

Now, with your tool or function integrations, Claude can:

  • Schedule things on your calendar
  • Book appointments
  • Look up weather, sports, or news
  • Fetch info from any REST API (...or follow your own business logic)

All you have to do is define the right tool in the tools array, and you and Claude are off to the races.

More resources

Inspired? I know I was! Here are some places to visit next for more ConversationRelay and Anthropic (including the earlier tutorials in this series):

Paul Kamp is Technical Editor-in-Chief at Twilio. He’s occasionally funnier than jokeapi.dev. (Or, at least, he thinks so.) You can contact him at pkamp [at] twilio.com.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.