Microsoft Foundry connectors
This guide shows you how to connect Microsoft Foundry agents to Twilio communication channels using the twilio-agent-connect-microsoft package. The package provides two connectors — Agent Framework (recommended) and Voice Live — that handle channel routing, memory context, and session management.
When you're ready to deploy to production, see Deploy to Azure Container Apps.
Before you begin, make sure you have:
- TAC SDK installed and configured with your Twilio credentials and a Conversation Configuration (see Add TAC to your agent prerequisites)
- Python 3.10+
- A Microsoft Foundry project with a deployed model (for example, GPT-4o)
- An Azure OpenAI API key for your deployed model
If you plan to use the Voice Live connector, you need a Microsoft Foundry Voice Live resource and a separate API key. See Voice Live connector for setup.
The twilio-agent-connect-microsoft package provides connectors for different Microsoft Foundry services.
| Connector | Use case | Install extra |
|---|---|---|
AgentFrameworkConnector | Build agents with Microsoft Agent Framework. Supports voice and SMS, session persistence, and custom tools. Recommended for most use cases. | agent-framework |
VoiceLiveConnector | Use Microsoft Foundry Voice Live for low-latency streaming inference over WebSocket. Voice only. | voice-live |
The Agent Framework connector is the recommended way to connect a Microsoft Foundry agent to Twilio channels. It supports both voice and SMS, with configurable session persistence.
pip install twilio-agent-connect-microsoft[agent-framework,server]
Add the following to your .env file alongside your existing TAC credentials.
1AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/2AZURE_AI_API_KEY=your_azure_openai_api_key3AZURE_AI_DEPLOYMENT_NAME=gpt-4o
The AgentFrameworkConnector takes a function that creates an agent for each voice call or SMS message. The connector handles memory retrieval, channel routing, and conversation cleanup.
1# Fix SSL certificate verification (must be before other imports)2import truststore3truststore.inject_into_ssl()45import os67from agent_framework.openai import OpenAIChatClient8from dotenv import load_dotenv9from pathlib import Path1011from tac_microsoft import (12TAC,13TACConfig,14SMSChannelConfig,15TACFastAPIServer,16AgentFrameworkConnector,17ConversationSession,18)1920load_dotenv(Path(__file__).resolve().parent.parent / ".env")2122# Azure AI client23client = OpenAIChatClient(24azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],25api_key=os.environ["AZURE_AI_API_KEY"],26model=os.environ.get("AZURE_AI_DEPLOYMENT_NAME"),27)2829# TAC setup30tac = TAC(config=TACConfig.from_env())3132SYSTEM_PROMPT = "You are a helpful customer service agent. Be concise and friendly."333435def create_agent(session: ConversationSession):36"""Return an Agent Framework agent for this conversation."""37return client.as_agent(38name="MyAgent",39instructions=SYSTEM_PROMPT,40)414243connector = AgentFrameworkConnector(44tac=tac,45create_agent=create_agent,46sms_config=SMSChannelConfig(memory_mode="always"),47)4849server = TACFastAPIServer(50tac=tac,51voice_channel=connector.voice_channel,52messaging_channels=[connector.sms_channel],53)5455if __name__ == "__main__":56server.start()
The connector automatically:
- Retrieves memory context from Conversation Memory and includes it in the first message (if memory retrieval is enabled)
- Routes responses to the correct channel (Voice or SMS)
- Cleans up agent resources when the conversation ends
You can customize the agent per channel, add tools inside your create_agent function, and pass hooks like on_message and on_error to the connector. This example adds:
- Channel-aware prompts: Different system prompts for voice and SMS.
- Custom tools: A
look_up_outage_toolalongside built-in TAC tools (create_memory_tool,create_knowledge_tool). on_messagehook: Prepends customer phone number and memory context to each message.on_errorhook: Returns channel-appropriate error messages.- Session persistence:
FileAgentSessionStorefor local development, with aCosmosDBAgentSessionStoreoption for production. - Conversation cleanup: Deletes session data when a conversation ends.
1# Fix SSL certificate verification (must be before other imports)2import truststore3truststore.inject_into_ssl()45import asyncio6import logging7import os8from pathlib import Path910from agent_framework.openai import OpenAIChatClient11from azure.identity.aio import DefaultAzureCredential12from dotenv import load_dotenv1314from tac_microsoft import (15TAC,16TACConfig,17TACFastAPIServer,18AgentFrameworkConnector,19FileAgentSessionStore,20ConversationSession,21VoiceChannelConfig,22SMSChannelConfig,23format_memory_context,24)25from tac_microsoft.agent_framework_tools import create_knowledge_tool, create_memory_tool2627load_dotenv(Path(__file__).resolve().parent.parent / ".env")2829logger = logging.getLogger(__name__)3031credential = DefaultAzureCredential()32client = OpenAIChatClient(33credential=credential,34azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],35model=os.environ.get("AZURE_AI_DEPLOYMENT_NAME", "gpt-4o"),36)3738tac = TAC(config=TACConfig.from_env())39knowledge_base_id = os.environ.get("TWILIO_KNOWLEDGE_BASE_ID")4041# Build the knowledge tool once at startup — it doesn't depend on session state.42knowledge_tool = (43asyncio.run(create_knowledge_tool(44tac,45knowledge_base_id,46description="Search the knowledge base for relevant information.",47))48if knowledge_base_id49else None50)5152# Channel-aware system prompts53VOICE_SYSTEM_PROMPT = """You are a customer service assistant on a phone call.54Keep responses clear, concise, and conversational.55Use plain text only — no markdown, no lists, no special formatting."""5657SMS_SYSTEM_PROMPT = """You are a customer service assistant over SMS.58Keep responses concise and formatted for text messaging.59Use short paragraphs. Bullet points are OK."""606162# Custom tool63def look_up_outage_tool(zip_code: str) -> str:64"""Check if there is a recent internet outage in a specific zip code."""65return f"No reported outages in {zip_code}. Service is operating normally."666768# Agent factory — called once per voice call, once per SMS message69def create_agent(session: ConversationSession):70prompt = VOICE_SYSTEM_PROMPT if session.channel == "voice" else SMS_SYSTEM_PROMPT7172tools = [create_memory_tool(tac, session), look_up_outage_tool, knowledge_tool]7374return client.as_agent(75name="OwlAgent",76instructions=prompt,77tools=[t for t in tools if t is not None],78)798081# Session persistence (FileAgentSessionStore for local development)82session_store = FileAgentSessionStore()8384# To use CosmosDB instead (for horizontal scaling), uncomment below:85# from tac_microsoft import CosmosDBAgentSessionStore86# session_store = CosmosDBAgentSessionStore(87# endpoint=os.environ["AZURE_COSMOS_ENDPOINT"],88# credential=os.environ["AZURE_COSMOS_KEY"],89# )909192def on_message(user_message, context, memory_response):93"""Customize the user message with context before sending it to the agent."""94prefix = f"[Customer: {context.author_info.address if context.author_info else 'unknown'}]\n"95return prefix + format_memory_context(memory_response, user_message)969798def on_error(error, context):99"""Return a channel-appropriate error message."""100logger.error("Agent error", extra={"conversation_id": context.conversation_id}, exc_info=error)101if context.channel == "voice":102return "I'm sorry, I'm having trouble right now. Please try again."103return "Sorry, something went wrong. Please try again or call us for help."104105106async def handle_conversation_ended(context: ConversationSession) -> None:107"""Clean up sessions when a conversation closes."""108await session_store.delete(context.conversation_id)109logger.info("Session cleaned up", extra={"conversation_id": context.conversation_id})110111112tac.on_conversation_ended(handle_conversation_ended)113114connector = AgentFrameworkConnector(115tac=tac,116create_agent=create_agent,117on_message=on_message,118on_error=on_error,119voice_config=VoiceChannelConfig(memory_mode="never"),120sms_config=SMSChannelConfig(memory_mode="always"),121session_store=session_store,122)123124server = TACFastAPIServer(125tac=tac,126voice_channel=connector.voice_channel,127messaging_channels=[connector.sms_channel],128)129130if __name__ == "__main__":131server.start()
Microsoft Foundry Voice Live provides low-latency streaming inference over WebSocket. The VoiceLiveConnector supports voice only (no SMS).
pip install twilio-agent-connect-microsoft[voice-live,server]
Add the following to your .env file alongside your existing TAC credentials. The endpoint is the hostname only — no https:// prefix.
1AZURE_VOICE_LIVE_ENDPOINT=your-resource.services.ai.azure.com2AZURE_VOICE_LIVE_API_KEY=your_voice_live_api_key3AZURE_VOICE_LIVE_MODEL=gpt-4o
1# Fix SSL certificate verification (must be before other imports)2import truststore3truststore.inject_into_ssl()45import os67from dotenv import load_dotenv8from tac.tools.base import function_tool910from tac_microsoft import (11TAC,12TACConfig,13TACFastAPIServer,14VoiceLiveConnector,15VoiceLiveConfig,16)1718load_dotenv()1920# Custom tools — use @function_tool to create TACTool instances21@function_tool()22def look_up_outage(zip_code: str) -> str:23"""Check if there is a recent internet outage in a specific zip code.2425Args:26zip_code: The zip code to check for outages.27"""28return f"No reported outages in {zip_code}. Service is operating normally."293031tac = TAC(config=TACConfig.from_env())3233config = VoiceLiveConfig(34endpoint=os.environ["AZURE_VOICE_LIVE_ENDPOINT"],35model=os.environ.get("AZURE_VOICE_LIVE_MODEL", "gpt-4o"),36api_key=os.environ.get("AZURE_VOICE_LIVE_API_KEY"),37instructions="You are a helpful customer service agent. Be concise and friendly.",38tools=[look_up_outage],39)4041connector = VoiceLiveConnector(42tac=tac,43config=config,44)4546server = TACFastAPIServer(47tac=tac,48voice_channel=connector.voice_channel,49)5051if __name__ == "__main__":52server.start()
Voice Live manages conversation state server-side through its WebSocket API, so you don't need local session management. When the model requests a tool call, the connector executes it and sends the result back through the WebSocket automatically.
To start from a working example instead of writing the server code yourself, clone the twilio-agent-connect-microsoft repository and run one of the included examples.
-
Clone the repository and install dependencies:
1git clone https://github.com/twilio/twilio-agent-connect-microsoft.git2cd twilio-agent-connect-microsoft3uv sync --all-extras -
Create the shared
.envfile from the template. All examples read from this file:cp getting_started/examples/.env.example getting_started/examples/.env -
Fill in your TAC credentials and the environment variables for the example you want to run:
Agent Framework (basic)Agent Framework (advanced)Voice LiveBasic example uses Azure OpenAI with API-key authentication.
1AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/2AZURE_AI_API_KEY=your_azure_openai_api_key3AZURE_AI_DEPLOYMENT_NAME=gpt-4o -
From the repository root, start the example server:
Agent Framework (basic)Agent Framework (advanced)Voice Liveuv run getting_started/examples/agent_framework/basic.py -
Expose your local server with ngrok so Twilio can reach it, then configure your Twilio webhooks to point at the ngrok domain. For detailed steps, see Expose your local server with ngrok and Configure Twilio webhooks.
- Start your server and make sure ngrok is running.
- Call your Twilio phone number and verify that the voice channel works.
- If you used the Agent Framework connector, send an SMS to your Twilio phone number and verify that you receive a response.
- If you configured Conversation Memory, verify that responses include personalized context from the customer's profile.
- Deploy to Azure Container Apps: Deploy your TAC application to production on Azure.
- Add TAC to your agent: Add custom tools and knowledge search.
- Troubleshooting: Common issues and solutions.