Build Persistent Customer Memory with Twilio Agent Connect and Conversation Intelligence

Time to read:

August 14, 2026

Written by

Simran Aishwarya

Twilion

Reviewed by

Dhruv Patel

Twilion

AI generated summary

TAC and Flex integration streamlines AI-to-human escalation.
Conversational Intelligence extracts customer preferences for future interactions.
Twilio Memory API enables persistent customer profiling without a manual database.

This summary was generated by AI and reviewed by the Twilio team.

Build Persistent Customer Memory with Twilio Agent Connect and Conversation Intelligence

Introduction

What if your AI assistant already knew a customer's preferred language, favorite car model, and color before they said a word? That's exactly what this tutorial builds.

You'll integrate Twilio Agent Connect (TAC) with Twilio Flex to handle AI-to-human escalations, and layer on Twilio Conversational Intelligence to extract customer preferences from conversation summaries and persist them as traits in Twilio's Conversation Memory. The next time that customer reaches out, the AI assistant picks up right where the relationship left off.

By the end of the tutorial, you'll have a working escalation pipeline, an OpenAI-powered trait extractor, and a memory profile that survives across sessions with no database of your own to manage.

How it works

Here's the end-to-end flow before you write a single line of code:

A customer contacts your Twilio number (voice or SMS), and TAC routes them to the AI assistant.
The AI assistant handles the conversation. If the customer asks to speak to a human, TAC triggers a Twilio Studio handoff flow and routes them to a live Flex agent.
The conversation ends, and Conversational Intelligence generates a plain-language summary of the exchange.
A webhook fires your Twilio Function. The function calls OpenAI, which extracts structured preferences (model, color, language, etc.) from the summary and writes them as traits to the customer's Memory profile via the Twilio Memory API.
The next time the customer calls, TAC fetches the Memory profile at session start and injects those traits into the system prompt, so the AI responds in their preferred language automatically, without being asked.

Prerequisites

To deploy this project, you will need:

Python 3.10 or newer installed on your local machine (tested on Python 3.14.4).
A Twilio account with an active, SMS- and voice-capable phone number.
A Twilio Flex instance configured and running.
An OpenAI account alongside a valid API key.
A Twilio Functions service (you'll create this in Step 7).
ngrok installed to expose your localhost to Twilio's webhooks.
A phone to place test calls and verify the escalation workflow.

Build the app

Step 1: Project setup and initialization

Open your terminal, navigate to your project directory, and install the required SDKs and dependencies:

pip install twilio-agent-connect openai python-dotenv

Create a .env file in the same folder. Open .env in your preferred text editor and add the following fields:

# Twilio core credentials
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token_here
 
# Twilio API credentials (Generate via 1Console >Settings > Account Settings > API keys & auth tokens > create a standard API key.)
TWILIO_API_KEY=SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_API_SECRET=your_api_secret_here
 
# Third-party integrations
OPENAI_API_KEY=sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
# Application configuration
TWILIO_PHONE_NUMBER=+1XXXXXXXXXX
TWILIO_LOG_LEVEL=DEBUG
 
# Fill these in as you complete the steps below
TWILIO_CONVERSATION_CONFIGURATION_ID=
TWILIO_VOICE_PUBLIC_DOMAIN=
TWILIO_STUDIO_HANDOFF_FLOW_SID=
TWILIO_MEMORY_STORE_SID=

The empty variables at the bottom fill in as you progress through the next steps.

Step 2: Provision Flex and configure Conversations (classic)

If you don't have a Flex instance running yet, follow the Twilio Flex Account Setup Guide to set up Flex in your account. This automatically creates a default Flex Conversation Service behind the scenes.

If you want Flex to access your existing phone numbers and resources, review the public beta terms, check the agreement box, and click Add Flex.

Next, create an address rule so Twilio knows what to do when a text hits your phone number:

In the Twilio Console, navigate to Conversations (Classic) and select Addresses.
Click Create Address, or select your active Twilio number from the list.
Under Address Configuration, set Auto-create a Conversation to Yes.
For Conversation Service, select Flex Conversation Service from the dropdown.
Quick sanity check: make sure this matches the service listed under Conversations (Classic) > Settings > Default, or your routing will break.
For the Want to set up an integration? prompt, select No. A hardcoded webhook is not needed here, because your Studio flow and Conversation Orchestrator will handle the handoff logic dynamically.
Save your changes.

Step 3: Configure the Twilio Conversation Orchestrator

The Conversation Orchestrator handles interaction traffic and acts as the brains behind your routing and profiling.

Open your Twilio Console and navigate to the Conversation Orchestrator dashboard.
Create a new configuration and provide a distinct Conversation configuration name.
For Conversation grouping, select Group by profile.
Under the Webhook section, leave Webhook URL and HTTP Method empty for now. You'll return to update these once your tunnel is live in Step 5.
Do not select any Twilio phone numbers in Messaging & Chat traffic.
Check the box to enable Connect Conversations (Classic) service or Flex, then select Flex Conversation Service from the dropdown. This establishes the routing pipeline to your Flex instance for human agent handoffs.

Conversation Orchestrator configuration with Flex Conversation Service selected.

7. Do not enable Set up automatic capture for Voice traffic.

8. Enable Conversation memory. For an initial implementation, click + Create new memory store and ensure both Observations and Summaries are toggled on.

Conversation Memory enabled with Observations and Summaries toggled on.

9. On the Summary page, click Create Conversation Configuration.

10. Copy the Conversation configuration ID of the Conversation Orchestrator and add it to your .env file: TWILIO_CONVERSATION_CONFIGURATION_ID=conv_configuration_xxxxxxxxxxxxxxxxxxxxxxxxxx

11. Copy the Memory SID of the memory store, and also add it to your .env file: TWILIO_MEMORY_STORE_SID=mem_store_xxxxxxxxxxxxxxxxxxxxxxxxxx

For existing TAC users: if you have an existing Conversation Orchestrator setup with phone numbers enabled under Channel Traffic, leaving your Twilio number assigned in both places will cause a conflict. The Conversation Orchestrator will try to process the incoming SMS traffic twice. To make this work seamlessly with Twilio Agent Connect, remove the Twilio phone number as a sender from all channels under Automatically captured traffic in Channel Traffic.

Channel Traffic settings with the phone number removed from automatically captured channels.

Step 4: Establish Conversation Intelligence rules

Conversational Intelligence synthesizes summaries and fires webhooks when conversations end. You'll create two rules, but Rule 2 requires your Twilio Function URL, which you won't have until Step 7. Create Rule 1 now and return for Rule 2 in Step 8.

Go to Products & Services > Conversation Orchestrator > Conversation Intelligence > Intelligence configurations.
Link your new intelligence profile directly with the Conversation Orchestrator you built in Step 3.
Click your newly created intelligence configuration to view its parameters.
On the Details tab, locate the Rules section and click Create Rule.
Select Summary & Next-Best-Response as your primary language operator and click Next.
Set the activation trigger to At conversation end and leave the Webhook action blank.
In the Add Context section, select Enable Conversation Memory for this rule.
Save your changes.

You will create Rule 2 after Step 7, once your Twilio Function is deployed and you have its URL. Rule 2 uses the same settings but triggers After conversation ends and includes your Function URL as a webhook action. A reminder appears at the end of Step 7.

Step 5: Expose your local environment via ngrok

Twilio Agent Connect runs on port 8000 locally. You must expose this port to the web so Twilio can send incoming event payloads.

Open a fresh terminal window and start an HTTP tunnel on port 8000:

ngrok http 8000

Ensure your ngrok configuration targets port 8000 to match the TAC runtime port.

ngrok outputs a dynamic forwarding address similar to: https://a1b2-34-56-78.ngrok-free.app

Return to your local .env file and set TWILIO_VOICE_PUBLIC_DOMAIN to your public ngrok URL, without the https:// prefix: TWILIO_VOICE_PUBLIC_DOMAIN=a1b2-34-56-78.ngrok-free.app

Next, update your webhook endpoints:

For Voice traffic: navigate to Numbers and Senders > Overview, select your number, and set the Voice webhook to your ngrok URL with the /twiml endpoint. Example: https://a1b2-34-56-78.ngrok-free.app/twiml.
For the Orchestrator: go to the Conversation Orchestrator, edit your configuration, and set the Webhook URL using the /webhook endpoint. Example: https://a1b2-34-56-78.ngrok-free.app/webhook.

Step 6: Link the Studio flow for human handoff

This step ensures that when a handoff is triggered, the call or message routes through Twilio Studio to a Flex agent.

Create a flow from a template. In the Twilio Console, navigate to Studio > Flows. Click Create flow, then select From template from the dropdown.
Select the handoff template. Scroll to the bottom and select Twilio Agent Connect - Human Handoff. This template contains the pre-configured logic for TAC escalations.
Configure Flex routing. Open the flow editor and click the send_to_flex widget. Under Workflow settings, select your desired Flex workflow (for example, Assign to Anyone).
Capture the Flow SID. Save and publish the flow. Note the Flow SID (it starts with FW).
Finalize your .env .Add this SID to your environment file: TWILIO_STUDIO_HANDOFF_FLOW_SID=FWXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Step 7: Create the Twilio Function

Create the Function service:

Log in to your Twilio Console.
Navigate to Functions and Assets > Services.
Click Create Service.
Name your service dynamictrait and click Next.

Create a public path:

Inside your new dynamictrait service, click the Add + button at the top left.
Select Add Function from the dropdown.
Change the path name to /trait.
Set the visibility dropdown next to the path to Public. This allows external requests to reach your function.

Configure environment variables and dependencies:

In the service sidebar, click Dependencies.
Under Environment Variables, check the box Add my Twilio Credentials (ACCOUNT_SID and AUTH_TOKEN) to ENV. This automatically injects your core Twilio credentials into the function's environment.
Manually add the following custom keys and their values: TWILIO_MEMORY_STORE_SID and OPENAI_API_KEY
Under Dependencies, set the Node.js Version dropdown to the latest available version.
Add the following npm packages, each set to latest: twilioaxios and openai
Click Save at the bottom of the page.

Add your Function code:

Replace the code inside your /trait function with the following:

const axios = require('axios');
const { OpenAI } = require('openai');
exports.handler = async function(context, event, callback) {
    try {
        const openai = new OpenAI({ apiKey: context.OPENAI_API_KEY });
        const summary = event.operatorResults?.[1]?.result?.text;
        const conversation_sid = event.conversationId;
        const parti_obj = event.operatorResults?.[0]?.executionDetails?.participants || [];
        const extractmem = parti_obj.filter((x) => x.type === "CUSTOMER");
        const mem_profile = extractmem[0]["profileId"];
        console.log("The Memory profile of customer : ", mem_profile);
        console.log("The Summary of ", conversation_sid, " : ", summary);
        async function updateTrait({ key, value }) {
            try {
                console.log("save", key, value);
                const accountSid = context.ACCOUNT_SID;
                const authToken = context.AUTH_TOKEN;
                const mem_store = context.TWILIO_MEMORY_STORE_SID;
                const url = `https://memory.twilio.com/v1/Stores/${mem_store}/Profiles/${mem_profile}`;
                const requestmade = await axios.patch(url, { traits: { Preferences: { [key]: value } } },
                    {
                        auth: {
                            username: accountSid,
                            password: authToken,
                        },
                        headers: {
                            'Content-Type': 'application/json'
                        }
                    }
                );
                console.log("Twilio Memory updated successfully:", requestmade.data);
            }
            catch (error) { console.log("Error:", error.message); }
        }
        const tools = [
            {
                type: "function",
                name: "update_trait",  
                description: "Save user preferences as traits",
                parameters: {
                    type: "object",
                    properties: {
                        key: { type: "string", enum: ["model", "color", "country", "language"] },
                        value: { type: "string" }
                    },
                    required: ["key", "value"],
                    additionalProperties: false
                }
            }
        ];
        const response = await openai.responses.create({
            model: "gpt-4.1-mini",
            input: [
                {
                    role: "system",
                    content: "Extract user preferences (model, color, country, language from the summary. Call update_trait once for each preference found."
                },
                { role: "user", content: summary }
            ],
            tools,
            tool_choice: "auto"
        });
        console.log(response.output);
        for (const item of response.output) {
            if (item.type === "function_call" && item.name === "update_trait") {
                const args = JSON.parse(item.arguments);
                console.log(args);
                await updateTrait(args);
            } else {
                console.log("item type is not of function_call");
            }
        }
        return callback(null, "ok");
    }
    catch(error){
        console.log(error); 
        return callback(error);
    }
}

Once you've pasted the code, click Save, then click Deploy All to bring your serverless function live.

After deploying, copy your function's public URL. It looks like: https://dynamictrait-XXXX.twil.io/trait

You'll need this URL in the next step.

Step 8: Complete Conversation Intelligence — add Rule 2

Now that your Twilio Function is live, return to your intelligence configuration and create the second rule.

Go to Conversation Orchestrator > Conversation Intelligence > Intelligence configurations.
Open the same configuration you worked on in Step 4.
Under Rules, click Create Rule.
Select Summary & Next-Best-Response as the language operator.
Set the activation trigger to After conversation ends.
Under Webhook action, paste your Twilio Function URL: https://dynamictrait-XXXX.twil.io/trait
In Add Context, enable Conversation Memory.
Save the rule.

Two Intelligence rules configured under the Conversation Intelligence service.

You'll now have two rules under your Intelligence Configurations service.

Step 9: Set up Memory trait groups

Before testing, define the trait structure where customer preferences will be stored.

In the Twilio Console, navigate to Conversation Memory. Select your memory store and choose Traits.
The default trait group is Contacts traits. Create a new trait group called Preferences to store customer preferences.
Click + Add trait group and name it Preferences.

Add trait group dialog in Conversation Memory.

New Preferences trait group listed alongside Contacts traits.

Click + Add trait to add each trait you want dynamically populated from the customer's interaction summary.

Add trait dialog for the Preferences group.

Add one trait for each preference you want to capture. For this demo (a car company), add:
model
color
country
language

Preferences trait group populated with model, color, country, and language traits.

Step 10: Implement the core TAC backend logic

Now that your Twilio infrastructure is fully mapped out, look at the Python backend script that runs the AI assistant and coordinates the handoff. The base code is available in the https://github.com/twilio/twilio-agent-connect-python.

The version below extends it with Memory retrieval and language-preference support.

Create a new file in your folder, give it a name ex : agent_handoff.py, and copy paste the code given below.

from typing import Any
from agents import Agent, Runner, set_tracing_disabled
from dotenv import load_dotenv
from tac import TAC, TACConfig
from tac.channels.sms import SMSChannel, SMSChannelConfig
from tac.channels.voice import VoiceChannel, VoiceChannelConfig
from tac.models.session import ConversationSession
from tac.models.tac import TACMemoryResponse
from tac.server import TACFastAPIServer
from tac.tools.handoff import create_studio_handoff_tool
load_dotenv()
set_tracing_disabled(True)
tac = TAC(config=TACConfig.from_env())
# Verify the handoff-specific env var is set.
if not tac.config.studio_handoff_flow_sid:
    raise RuntimeError(
        "TWILIO_STUDIO_HANDOFF_FLOW_SID is required to run the handoff example. "
        "Set it in your .env (see .env.example)."
    )
SYSTEM_INSTRUCTIONS = (
    "You are a customer service agent speaking with a user over voice or SMS. "
    "Keep responses short and conversational — a sentence or two. "
    "Do not use markdown, asterisks, bullets, or emojis; your words will be "
    "spoken aloud or sent as plain text. "
    "If the user asks to speak with a human, or if you cannot resolve their issue, "
    "use the handoff tool to transfer them to a human agent."
)
HANDOFF_ATTRIBUTES = {
    "department": "support",
    "priority": "normal",
}
conversation_history: dict[str, list[Any]] = {}
async def handle_message_ready(
    user_message: str,
    context: ConversationSession,
    memory_response: TACMemoryResponse | None,
) -> str:
    conv_id = context.conversation_id
    if conv_id not in conversation_history:
        conversation_history[conv_id] = [
            {"role": "system", "content": SYSTEM_INSTRUCTIONS}
        ]
    conversation_history[conv_id].append({"role": "user", "content": user_message})
    handoff_tool = create_studio_handoff_tool(
        tac, context, attributes=HANDOFF_ATTRIBUTES
    )
    SYSTEM_INSTRUCTIONS_modified = SYSTEM_INSTRUCTIONS
    if memory_response:
        memory_sections = memory_response.build_memory_prompts()
        if memory_sections:
            SYSTEM_INSTRUCTIONS_modified = SYSTEM_INSTRUCTIONS_modified + (
                "\n\n" + "\n\n".join(memory_sections)
            )
        profile = context.profile
        print(profile)
        if profile and profile.traits:
            language = profile.traits.get("Preferences", {}).get("language")
            print("User Preffered Language :", language)
            uselang = f"Switch to language {language} for further communication. You can change the language if you are asked to change language or if the customer speaks in a different language."
            SYSTEM_INSTRUCTIONS_modified = SYSTEM_INSTRUCTIONS_modified + uselang
    agent = Agent(
        name="Customer Service Agent",
        instructions=SYSTEM_INSTRUCTIONS_modified,
        tools=[handoff_tool.to_openai_agents_sdk_tool()],
    )
    history = conversation_history.get(context.conversation_id, [])
    agent_input = history + [{"role": "user", "content": user_message}]
    result = await Runner.run(agent, agent_input)
    conversation_history[context.conversation_id] = result.to_input_list()
    return result.final_output_as(str)
voice_channel = VoiceChannel(tac, config=VoiceChannelConfig(memory_mode="always"))
sms_channel = SMSChannel(tac, config=SMSChannelConfig(memory_mode="always"))
tac.on_message_ready(handle_message_ready)
if __name__ == "__main__":
    server = TACFastAPIServer(
        tac=tac, voice_channel=voice_channel, messaging_channels=[sms_channel]
    )
    server.start()

Test the end-to-end flow

Run the script from your terminal:

python agent_handoff.py

First interaction — new customer

Call or text your active Twilio number. The AI assistant picks up and responds in English (the default, since there is no memory yet).
Have a conversation about a car — for example:
"I'm looking for a blue Model X. Do you have it in stock?"
"What red sedans do you have available?"
Ask to be transferred: "Can I speak to a human agent?". TAC triggers the Studio handoff flow, and the conversation routes to a live Flex agent.

SMS conversation with the AI assistant leading into a human handoff request.

AI assistant handing the caller off to a Flex agent.

Flex agent receiving the handoff task in the Flex UI.

Once the conversation closes, Conversational Intelligence generates a summary. Rule 2 fires your Twilio Function, which calls OpenAI to extract preferences and writes them to the customer's Memory profile.

To verify:

In the Twilio Console, go to Conversation Memory > Memory Store > Profiles.
Find your customer's profile.
Open Preferences Traits. You'll see the extracted values — for example, color: grey, model: Model X.

Second interaction — returning customer

Call or text the same Twilio number from the same phone.
TAC fetches the customer's Memory profile at session start and injects the traits into the system prompt.
Notice that the AI already knows the customer's preferences. If language was captured, the AI responds in that language without being asked.

AI assistant greeting the returning customer in their preferred language.

What happened behind the scenes

When the first conversation ended, this chain ran automatically:

The conversation ended.
Conversational Intelligence generated a plain-language summary.
Rule 2 fired a POST to your Twilio Function /trait.
The function extracted structured traits via OpenAI .
Traits were written to the customer's Memory profile via the Memory API.
On the next call, TAC read those traits and injected them into the system prompt.

No manual steps. No database to manage. The customer's preferences persist across sessions automatically.

Conclusion

In this tutorial, you built a full AI-to-human escalation pipeline with Twilio Agent Connect and Twilio Flex, and extended it with persistent, personalized memory. Conversational Intelligence extracts meaning from conversation summaries, OpenAI structures that meaning into traits, and Twilio Memory makes those traits available on every future interaction. The result is an AI assistant that remembers who it's talking to.

Simran Aishwarya is a Developer Support Engineer at Twilio who specializes in communication platforms, backend systems, and technical problem solving. Her interests span Python, JavaScript, cloud technologies, analytics, and DevOps, and she enjoys building practical solutions and continuously expanding her engineering skill set.

Related Resources

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.

Build Persistent Customer Memory with Twilio Agent Connect and Conversation Intelligence

Step 3: Configure the Twilio Conversation Orchestrator

Related Posts

Related Resources