Use ConversationRelay with the Open Source Agent Builder Flowise: A Multi-Agent Example with Voice

Use ConversationRelay with the Open Source Agent Builder Flowise: A Multi-Agent Example with Voice

Here at Twilio, we’ve seen customers looking for approaches for multi-agent solutions. Whether they’re looking to enable their customer support flow or their marketing lead qualification, they want to have specialized agents who can work together to answer questions or resolve customer issues.

In this post, we’ll show you how to set up and orchestrate a multi-agent solution to handle scheduling and provide information over the voice channel using Twilio Voice and ConversationRelay. We used Flowise, a visual agent builder, to put together this tutorial, and we are also open sourcing the package used in this approach so that you can replicate this in your environment for your own customer flows.

Let’s get started with the build!

Architecture of a multi-agent experience built with Flowise

Flowise makes it straightforward to build a multi-agent workflow. Here’s a sequence diagram of what we’ll be building.

In this demo, a Condition Agent using a mini model is used to quickly determine user intent. The 3 scenarios in our example are:

• Scheduling: Customer wants to book, schedule, or reserve a class, training session, appointment or pricing, plans, joining, cancelling, or upgrading their membership
• Fitness Guidance: Customer needs workout advice, exercise recommendations, or training guidance
• General Info: Customer asks about gym hours, location, amenities, or general services

Each scenario is then connected to an Agent, which is a ReAct wrapper that can execute a RAG pipeline with its own tools and knowledge.

Here is a quick video that shows you the end to end experience we will be enabling through this approach.

Prerequisites

Before you can build our sample app, you’ll need to sign up for accounts and have a few things ready.

• A Twilio Account (you can sign up here for free)
  • A Twilio Phone number with Voice capabilities
  • (Optional) A Twilio phone number with Messaging capabilities
• Flowise Account
• Twilio CLI with the Serverless Plugin
• API Key to your fav LLM model
• Node.js (Version 20+)
• Ngrok (guide: https://ngrok.com/docs/getting-started/)
• Fly.io (if you want to host remotely)

Once those are set up or installed, you’re ready for the build!

Build a multi-agent experience with Flowise

Now, let’s walk through building customer experience flows with Flowise. First, we’ll build some fitness agents on Flowise. We’ll then set up a local server to host our logic and interface with Twilio Voice using ConversationRelay. Then, finally, you’ll be able to call and interact with the agents!

Build agents on Flowise

In this step, we’ll walk through working in Flowise. We’ll create our agents in the Flowise dashboard, then grab our API Key to work with ConversationRelay.

Getting started

1. Clone the repo https://github.com/twilio-samples/cr-flowiseai and run npm install.
2. cd sample-tools, then run npm install
3. Copy the sample .env file: cp .env.example .env
4. Visit https://flowiseai.com/, and either sign up for an account (if you haven’t yet!), or log in
5. Go to the Credentials tab and create a new credential for the LLM model of choice, pasting in your API key.
6. Visit the API Keys tab and copy the API key. Paste it in the appropriate place in your .env

Great! Now, let’s create the Tools we’ll need to perform some of the other actions we’d like our agents to do.

Deploy Twilio Functions and create Tools

1. Go into the sample-tools directory from the repo you cloned
2. Set up Twilio CLI alongside the Serverless plugin if you haven’t already - instructions are available in the Readme in the sample-tools directory
3. Run npm run deploy to deploy the tools to your Twilio Functions
4. Copy the Twilio Functions URL into the .env
5. Now run ./tools/ deploy.js to deploy the Flowise Tools

Set up your agents

1. Switch back to the Flowise Website.

2. Visit Variables and create a static variable called customerPhoneNumber – set any value you want. This will be overwritten at runtime.

3. Go to Agentflows and create a new Flow – call it VoiceAgent.

4. Create the Flow from the screenshot above.

5. For the Conditional Agent configuration, set:

• Instructions

• Input

• Create 3 Scenarios
  • Scheduling: Customer wants to book, schedule, or reserve a class, training session, appointment or pricing, plans, joining, canceling, or upgrading their membership
  • Fitness Guidance: Customer needs workout advice, exercise recommendations, or training guidance
  • General Info: Customer asks about gym hours, location, amenities, or general services

6. For the Scheduling Agent

• Choose the LLM model you defined previously
• For the messages, set the System role, and for the content set Customer Contact Number: {{$vars.customerPhoneNumber}}
• Now, add 4 Tools: select Custom Tool and for the parameters, choose each of the Tool_ that were created from the previous step.
• For the Input Message set:

7. For the Fitness Agent

• Choose the LLM model you defined previously
• For the messages, set the System role and for the content, set Customer Contact Number: {{$vars.customerPhoneNumber}}
• For the Input Message set:

8. For the General Info agent

• Choose the LLM model you defined previously
• For the messages, set the System role, and for the content, set Customer Contact Number: {{$vars.customerPhoneNumber}}
• For the Input Message set:

• Click the </> icon on the top right corner and make sure the default API key is selected
• Grab the FLOWISE_CHATFLOW_ID (from the URL) and update your .env file
• Visit API Keys on the Flowise homepage to grab the FLOWISE_API_KEY

Deploy locally

To test that everything is set up correctly, let’s run the application locally.

Get started by cloning the app with git clone https://github.com/twilio-samples/cr-flowiseai if you haven’t yet. Then, cd cr-flowiseal and run:

1. ngrok http 3030 (or the port your dev server will run on)
2. Now, in a text editor or IDE, update your .env (cp .env.example .env) and add your ngrok domain as the DOMAIN variable (omit the scheme, or the “https://”). Paste in the two Flowise variables from the above section.
3. Optionally, change the PORT variable to match how you changed step 1
4. Then, in a different terminal than where ngrok is running, run npm run dev
5. Visit the Twilio Phone Numbers page and set the A call comes in -> Webhook to https://ngrok-domain.app/twiml on a number you’d like to call

Now go ahead and call your number!

If everything worked correctly, you should be able to chat with the Voice Agent via ConversationRelay. Try asking about classes or ask it for fitness advice, and see how it works.

Deploy to the cloud with Fly.io

We have some instructions to use Fly.io to host your application. It supports long-running WebSocket connections and is free (when you sign up, you’ll have to add your credit card, but you can deploy free servers for testing purposes).

This repo includes the configuration for easily deploying the application to Fly.io. All the configurations are located in the fly.toml file.

• Sign up for Fly.io and install/setup their CLI.
• Create a new Fly App: fly apps create cr-flowiseai
• Grab the domain name of your newly created Fly app and update your local .env with the DOMAIN value.
• Set up the secrets by running fly secrets import < .env
• Deploy the application: fly deploy
• View the logs: fly logs

You can now set the A call comes in -> Webhook of your Twilio number to point to DOMAIN/twiml.

Just like when you tested locally, try calling the number. Exercise the different agent paths to ensure it’s still working – and go ahead and close your laptop lid, it’ll keep working!

Setting up a multi-agent solution with Flowise, ConversationRelay and Twilio Voice

Congratulations! You’ve now built a multi agent solution you can quickly tailor to your needs. You can extend this build with knowledge specific agents to address complex scenarios – for example, customer support across domains, or specialist agents with access to advanced tools.

In our next post with Flowise, we’ll show you how to extend the solution with tool calling and variables you can define from within Flowise’s Dashboard. We’ll also show you one way to deploy your solution using Fly.io. Happy chatting!

Additional resources

• ConversationRelay Docs
• Flowise Docs
• Latency Extension for A2A

About Twilio Forward

Twilio Forward focuses on Horizon-3 initiatives focused on driving step-change innovation that empowers builders and unlocks Twilio’s next era of growth. As an incubation lab, we explore bold new ideas, from the most advanced, almost unimaginable technologies to emerging solutions that address today’s real-world challenges. Our mission is to push boundaries, reimagine what’s possible, and build what comes next.

Rikki Singh is a product and engineering leader based in Bay Area, California. At Twilio, she leads the Emerging Technology and Innovation group Twilio Forward. Outside of work, Rikki enjoys hiking and camping with her husband and toddler.

Kousha Talebian is a Principal Engineer from Vancouver, BC, working on the Emerging Technology and Innovation team. You can reach him at ktalebian [at] twilio.com . Outside of work, Kousha enjoys running with his dog and experimenting with various cuisines from around the world.

Paul Kamp is the Technical Editor-in-Chief of the Twilio Blog. You can reach him at pkamp [at] twilio.com