Contact sales Start for free
Contact sales Start for free

What did you say to me? Instrumenting User Insights for your AI Copilot

Time to read:

March 26, 2024
Written by
Vaith Schmitz
Twilion
Kevin Niparko
Twilion

What did you say to me? Instrumenting User Insights for your AI Copilot

AI copilots are the new app UX. 

Whether you’re building a chat bot or automated voice systems, your entire application is now only a prompt away from your users.

But how are your users interacting with these new model-powered experiences? And how can you take the ephemeral context from a chat window and use it to personalize the customer experience going forward?

For developers shipping LLM-powered copilots, there’s a new challenge to understand customers’ interactions with these automated systems. 

In this post, we’re going to show you how to deploy an AI Copilot experience with advanced instrumentation that allows you to analyze your Copilot performance and persist user context to drive the customer experience across sales, support, and marketing channels.

A screenshot of a messaging or chat application interface showing a sequence of tracked events such as message sent, conversation started, custom component loaded, stock purchased, message received, accompanied by timestamps, and a small info box titled "What's the App" at the bottom.

Instrumenting your AI Copilot with Twilio Segment

In this example, we’ll use the Vercel AI SDK and Chatbot template as a starting point. The Vercel AI SDK offers a twist on the traditional Copilot interface: rather than the model responding exclusively with text, the model can respond with text OR interactive components from your application to provide a richer user experience. An introduction to React Server Components (RSC) can be found in the Next.js docs.

We instrument tracking with the Twilio Segment node.js SDK to collect user and copilot behavior. Twilio Segment has integration capabilities for a wide range of other languages, so your analytics instrumentation can span different languages and implementations. We’re utilizing OpenAI’s GPT-3.5 for this showcase, but you can bring your own (customized) LLMs including Anthropic, Hugging Face, and more.

Note: While we’re skipping sign-ups and logins for brevity, it’s best practice to identify your copilot users with first-party data as soon as possible. 

Twilio Segment Setup

If you don’t have a Twilio Segment account yet, you can start by creating a new free account here. Once you’ve created your account, create a node.js source in Twilio Segment and note the write key, you will need this later to set up your tracking in the app. 

A screenshot of a user interface showing a list of data sources within a catalog, with a search box filtered by the keyword "model," featuring a highlighted data source icon and various categories on the left sidebar.

Step 1: Copilot Template Setup

To get started with your copilot setup, follow these steps:

The following commands get your app ready and deployed on localhost:

pnpm / npm install
pnpm / npm seed
pnpm / npm dev

A number of components come pre-configured with Twilio Segment tracking, with the library initiated in the analyticsInstance.ts singleton file. This makes it easy to integrate Twilio Segment on any component or route you want to use it on by simply importing.

import analytics from '@/app/analyticsInstance'

Step 2: Define your tracking plan

A clean copilot instrumentation starts by identifying your copilot’s event lifecycle. The Twilio Segment AI Copilot spec gives you an overview of relevant data points to track to understand your users’ interactions with your application. This spec is extensible to your specific use cases, a number of them will be covered in further detail in the implementation guide in this post. 

An infographic illustrating the lifecycle of an AI model within a system. It includes steps such as Identify, Conversation Started, Message Sent, Feedback Submitted, Message Received, Media Generated, Custom Component Loaded, Tool Invoked, and Conversation Ended, with icons and brief descriptions for each phase.

Event

Definition

Fields

Conversation Started

When a new conversation begins

conversationld

Message Sent

When a new message is added to a thread by user

conversationld,
messageld,
message_body

Media Generated

When an image or video is generated by the model

conversationld,
messageld,
type

Feedback Submitted

When a user rates a conversation or message

conversationld,
messageld,
rating

You can find the full Copilot spec here.

Step 3: Identify your user

You can track your users in both anonymous and known contexts. While we forgo this in the template, a quick way to do so is to identify unknown users using Twilio Segment’s anonymousId property in an identify call. Generate a unique ID through a method of your choice (e.g. uuid). You can then identify your users with the following Twilio Segment identify call. You can also enrich your user profile with properties if you’d like.

analytics.identify({
	anonymousId: YOUR GENERATED ID HERE,
	traits:{
key: value
...
}
})

Once your user has given you additional information (e.g. email, user id etc), create another identify call with a userId. This will likely happen on sign-up, and log-in will enable you to track behavior across sessions and devices.

analytics.identify({
	userId: USER ID HERE,
	traits:{
key: value
...
}
})

Ensure to pass either anonymousId or userId in all following track/page calls so your copilot interactions get attributed to the correct user.

Step 4: Tracking conversation metadata

Now that your users are identified, you can start instrumenting your copilot interactions. The first key event to track are conversation starts, allowing you to track stats like conversation duration and per conversation message count. The chat.tsx file in the starter template you have forked earlier generates a unique conversation ID for each new conversation, which is attached to all following tracked interactions that belong to the same conversation. The attached Vercel project stores conversation IDs in local storage as newChatId so you can easily grab them across your application.

analytics.track({
        userId: "123",
        event: "Conversation Started",
        properties: {
          conversationId: id
        }
      });
Conversation started

There are a number of extra events and properties you might want to track here, such as Conversation Ended events or specific models used if you have a multi-model setup. The Copilot AI spec has a number of extra events that might be worth exploring.

Step 5: Tracking standard prompts and responses

Tracking user prompts

The next important step is to understand how your customers interact with your copilot. To do so, it’s crucial to know what prompts they are using - Twilio Segment tracks this as a simple track event with Twilio Segment’s library, and passes in the message content as well as message and conversation IDs for full observability. The starter template collects your user’s prompt directly from the client in prompt-form.tsx, but you might wish to collect these from the server side as well using the same mechanism. Please ensure to pass in the relevant userId or anonymousId. The starter template uses hard-coded values for all of them.

analytics.track({
          userId: "123",
          event: "Message Sent",
          properties:{
            message_body:value,
            conversationId: window.localStorage.getItem('newChatId')
          }
        })
A screenshot of a command line curl request sending a tracking event to an API endpoint, including event details such as event type, message ID, timestamps, user ID, and write key.

Tracking copilot responses

Of course, you’re not just interested in your users’ prompts, but also the responses they receive from your models. Your app comes with tracking for your copilot’s response in the actions.tsx file of the starter template. The bot response is picked up from the server and sent into Twilio Segment, while simultaneously rendering to your user in their interface. You can of course enter additional properties to this event if you choose.

analytics.track({
          userId: '123',
          event: 'Message Received',
          properties: {
            message_body,
            conversationId: aiState.get().chatId,
          }
        })
A JSON-formatted data sample depicting a message received event, including context details, timestamps, message ID, conversation ID, content, user ID, and write key, used for data tracking or analytics.

Step 6: Tracking custom components and interactions

As copilots evolve from being text-only, Twilio Segment helps you track your customers' engagement with custom components that you’ve built or dynamically generated. Robust and data-agnostic tracking allows you to collect data on these interactions just like you would any other data point. The folks over at Vercel have done a great job implementing a number of custom components in the starter pack that are instrumented through Twilio Segment. The template ships with three pre-instrumented custom components, one for displaying multiple stocks, one displaying a stock and its chart price over the day, and one for purchasing stock directly from your copilot.

The first important event to track is how often and which of your non-standard (i.e. text/voice) components are called. This helps you understand which of your customizations are most popular across your user base and helps you focus on iterating them. The starter pack collects (in actions.tsx) the type of component loaded as well as additional information (stock symbols for a daily price chart, an array of symbols for multi-symbol display and purchase levels for purchase components) and sends this data into Twilio Segment.

// send custom component load to Twilio Segment
        analytics.track({
          userId: "123",
          event: "Component Loaded",
          properties: {
            type: 'Stock Price Chart',
            stock_symbol: symbol,
	conversationId: aiState.get().chatId
          }
        });
A screenshot of a financial stock chart showing the stock price of Apple (AAPL) with a current price of $145.2, an increase of 1.03%, and a line graph illustrating price movements over time, along with a timestamp of the latest data point (February 27, 4:59 PM EST).
A JSON payload for a stock price query event that includes the event type, message ID, timestamps, stock symbol, user ID, and write key, used for data tracking or analytics.

Twilio Segment can not only help you understand the custom components of your copilot, but more importantly, it also enables you to translate them into business relevant events, such as tracking stock purchases directly from custom components. Your app takes the user interaction on the custom purchase component and automatically translates it into a Stock Purchased event in Twilio Segment. 

// send purchase event to Twilio Segment
      analytics.track({
        userId: "123",
        event: "Stock Purchased",
        properties: {
          stock_symbol: symbol,
          amount: amount,
          total: amount*price
	conversationId: aiState.get().chatId
        }
      });
A screenshot of a financial trading interface showing a stock purchase calculator for Apple (AAPL). It displays a slider to select the number of shares, a total cost calculation (200 shares × $145.2 per share = $29,040.00), and a green "Purchase" button for executing the trade.
A media management interface showing a list of media items with their statuses and timestamps, and a media thumbnail labeled "Media" at the bottom.
A JSON payload for a stock purchase event including details such as stock symbol, message ID, timestamps, amount, user ID, and write key, used for data tracking or analytics.

Step 7: 360 Degree Copilot Observability

Twilio Segment’s event and data collection abilities don’t just stop there, but enable full business-centric observability for your copilots. This can also include understanding media your customers generate in your copilots, custom actions that are invoked, and data your copilots get back from your data warehouse. The newly launched AI Copilot spec gives you a baseline of mission critical events you can track with the help of Twilio Segment.

Turning Data into Action

Once collected, Twilio Segment gives you a breadth of tools to help further investigate your copilot’s user interaction by further enriching this data. You can create individual enrichments in computed traits, such as aggregating the number of times individual users have conversations with your copilot, or understanding which components they use most frequently, e.g. which stocks they interact with. 

A screenshot of a user interface showing a data analysis pipeline with step labels, conditions, timestamps, and trait previews, illustrating how data points like "User" are processed over time.
A user interface screen for creating or analyzing a trait related to user behavior, allowing the selection of event names, conditions, event properties, time windows, and options for including anonymous or historical users, with a trait preview on the right showing the name "User" associated with the stock symbol "AAPL".

By sending this data to one of Twilio Segment’s downstream partners, you can then activate users based on their interaction or send them to an analytics tool to gain further insight. 

One example of this can be pushing your copilot’s user interactions into an analytics tool like Mixpanel to further analyze stats like message frequency per conversation or how many messages your users exchange with your system until they complete a specific action (e.g. a stock purchase).

A dashboard displaying data visualizations, including a donut chart for message events per user with counts, and a line chart showing messages per conversation over time, with a media item labeled "Media" at the bottom right.

Your AI Copilot in the driver’s seat

By utilizing the outlined instrumentation strategies, not only are you finally understanding the black box of your customers’ conversations with your AI copilots, you are also able to leverage those insights to drive better, more personalized experiences at scale.

Importantly, Twilio Segment’s integration capabilities are not just single-sided. Our Profile API and Functions interfaces also allow you to send consistent, clean, and enriched customer data to your AI Copilot to drive more unique and tailored AI customer journeys.

Ready to see what Twilio Segment can do for you?

Get a demo

The Customer Data Platform Report 2025

Drawing on anonymized insights from thousands of Twilio customers, the Customer Data Platform report explores how companies are using CDPs to unlock the power of their data.

Get the report

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.