In the following video, Steve Tingiris, Founder and "chief dabbler" of Dabble Lab and Twilio Champion chatted with Nico Acosta, Director of Product for Autopilot at Twilio to talk about building digital assistants with Twilio Autopilot. Enjoy the interview transcript and watch the video for a deeper look at the tech.
The following interview can also be found on the Dabble Lab Youtube channel.
You can learn more about joining Steve and a community of supportive developers on the Twilio Champions page.
Introducing Nico Acosta & Twilio Autopilot
Steve: Hey, this is Steve with Dabble Lab, and this is another episode of Dabble With Experts. And today, I'm joined by Nico Acosta, who is the Director of Product and Engineering for Twilio Autopilot, and I've had the good fortune to work with Nico and Autopilot for a little bit, and so, today we get a chance to dabble a bit together, although he's going to be driving and showing us, what are you going to be showing us today, Nico?
Nico Acosta: Hey, Steve. So, we're going to be just playing around with Autopilot, seeing all the different features, all the things that you can do. There's a bunch of cool stuff here. So, we'll just be playing around, building different things, and showing you all the capabilities.
Steve: Excellent. And I know there's a lot of new stuff in recent weeks, so, yeah, let's jump in and see what there is.
Nico Acosta: Okay. So, I'm going to share my screen, if that's okay.
Steve: Yes, please do.
Nico Acosta: Great.
Steve: Got it.
Nico Acosta: So, a couple of things that I think is important to mention before, on how we think about conversational interfaces, and a little bit about Twilio and Autopilot. So, not sure if everyone has heard about Twilio before. We're a communications API platform that enables developers to build communication solutions using caller APIs, rather than old school telecomm hardware.
Nico Acosta: And Autopilot is our conversational AI platform, for developers to build intelligent assistants that they can deploy across any channel. So, you build once and you can deploy on any channel, which is kind of cool. And a lot of the things that we see when we see building intelligent assistants is an AI complete problem, probably one of the hardest problems out there, and a lot of what we try to do with Autopilot is give you all the power of machine learning while still making it super, super easy to program.
Machine Learning with Autopilot
Nico Acosta: And so, I'm going to walk through our InsuranceCo. So, this is an assistant that I have built here. First thing you'll notice about Autopilot is that we organize stuff with tasks, and tasks really represent kind of what the user wants to accomplish. So, greeting is our initiation task. Check claim status, file a claim, talk to someone, get an insurance quote, or do the NPS survey.
Nico Acosta: And then, one of the things we think a lot about is this separation between programming and training. These are kind of very different modes that you're in. So, when you click on train, you'll see here all the samples that are how this task is triggered. So, this is where the natural language understanding comes in. These are all the ways that somebody can say, "Hey, I want to know what's going on with my claim." And if you say it like, exactly what I said is not there, so, we can add it.
Nico Acosta: So, we'll say, "Hey, I want to know what's going on with my claim." Or if you imagine like saying this in a very different way, like, "Yo, where's my money?" Like, at the end of the day, you want to get paid. And you really want to capture all the different ways that somebody can say that.
Steve: And this will train the machine learning model, right? So, at some point, even if somebody phrases it in a way that's maybe not exactly the way that you put it in there, that will be mapped to the task as well?
Nico Acosta: Exactly, yes. You don't need to enter every single, it doesn't do a character by character match. It just builds a machine learning model. So, it will train it so that it can recognize things it has not seen before.
Steve: Got it.
Nico Acosta: And we go here to the natural language router, which built a, let's build a V2 of this model. And then, it'll take that sample that I added, and it'll make it in. But that's the, you can think about the training as how we get to a task, how is a task triggered, right? But then, there's a question of like, when we get to a task, what then?So, that's where programming comes in. So, let me, I'm going to create a new task.
Steve: So, the first half of it is sort of recognizing what the user wants, and then the second half is fulfilling what the user wants, the programming part?
Nico Acosta: Exactly. Imagine I want to cancel all of my policy. So, let's just follow the insurance, it is an auto policy, cancel policy. Let's go ahead and set a couple. And here you can do a multi-line. How do I cancel my policy? Just add these.
Nico Acosta: So now, we've got our brand new task, and we're going to go ahead and program. And here is where Autopilot's really, really powerful. Just get a little bit more screen real estate here. And so, you program the tasks with actions. And you can see all the actions here. You have say, to say something back to the user. So, it's like, I can help you cancel.
Nico Acosta: You can collect, and here in this right hand side, you'll see a bunch of different templates. So, collect for a yes/no question, you'll see, for multiple questions, asking multiple questions and validating the answers. So, let's look at this, let's edit in a bigger screen.
Nico Acosta: So, here you see essentially a question answering workflow, where you capture a bunch of data from the user, and you give it like first name, and you'll see the Twilio built-in field types, for like first name, date, time, number. You have like 18 different field types. So, if you copy this, you can paste it in your task. You can program this. You can do this in the actions bin, or you can do this with a URL. So, if you paste your server's URL here, you can render the actions directly from your application.
Steve: So, the action bin is going to save it there locally, and then the URL will allow you to call whatever endpoint you want to return a JSON format for the response back?
Nico Acosta: Exactly, exactly. But then, what is really neat, say you do a collect, right? Say, do you really want to cancel? And you have a URL right here, right? On complete, call this URL. And one of the cool things is that on complete, you fetch that URL, and you can return more JSON. So, you can keep going on a multi turn flow, in a multi turn conversation on a single task.
Nico Acosta: So, we see really three types of tasks. There are tasks that are system of record questions, like, what is the status of my claim? You know, assuming you know the customer, you go dip your database and come out with the status of the claim. Like, only you know the answer, right? And it's an answer that's like it's a system of record. You need to have a database with the status of the claims.
Nico Acosta: There's then like the knowledge questions type of answer, where it's like, will you insure a Lamborghini, yes or no, right? These are knowledge type questions. And the third one are data collection workflows, where these are multi turn conversations that go back and forth between the bot and the user, collecting different data, different pieces of data. And that's something I think Autopilot is doing in a unique way, where you don't need to create a task for every single turn of the conversation. You can do multiple turns of a conversation, because from a business logic perspective, you're still trying to get an insurance quote. So, let me show you, for example, how that works.
Steve: And these can be combined also, right? So, if I had a case where I wanted to use collect to collect information that was then going to be used for a query, to query a system of record, and respond with the results of the query, I could do that all as part of one task?
Nico Acosta: You absolutely could, yeah. So, let me show you that in action. So, here we have, let's go to insurance quote and program, and you'll see, we've put a URL here, right? And this URL points to a Twilio function, that you can think about it as kind of like a Lambda function. It's a Twilio runtime environment where we have all these auto insurance functions.
Steve: So like serverless, serverless code?
Nico Acosta: Serverless code, exactly. This could run here, it could run in your application server, it could run on Heroku, right?
Steve: Got it.
Nico Acosta: Wherever. So, if you look what we do, one of things we do is that we keep track of which state, which step we're in. So, auto insurance is a multi step task. So, we just keep track and implement the task step by step. So, great, I can help you with your auto insurance. I'll need to ask you a couple of questions. And then, here we go, we remember the state, the step, sorry, and we move it to one. Remember is really cool, because remember helps you build context on the conversation, and you can [inaudible 00:13:21] to remember any variable that is relevant for that session.
Steve: Okay, so that was a question I was going to ask. So, the scope of that is the session?
Nico Acosta: The scope of that is the session.
Steve: Got it.
Nico Acosta: So, you're not wary that you're going to switch to another task, and your data is going to get wiped out. Here we have a say. So, here's kind of, we're combining actions. We have a remember, we have a say, and we have a collect, asking us for the make of the car, the model, the year, and an email. And then, it sends it back to the same function, right? So, when it's done with that flow, it sends it back here.
Nico Acosta: This captures the step one, there's great hard coded stuff here, but you can imagine these are variables. And then, mark the step two, say the message, and when you look at the end, it's like we're done, we sent you the quote to the email, is there anything else I can help you with, and you essentially tell Autopilot that you're expecting an answer, saying listen equals true.
Nico Acosta: So, that is an example of an implementation of a multi step task. So, you don't need to create like, insurance step one or step two, you just create one task, and it's fairly dynamic how you implement it. It's like, the world is very dynamic. You can imagine you asking different questions depending on the type of car, right? Or you ask, prompt for the state, and then if it's California, you ask some questions, if it's Florida, you ask different questions. Who knows, right? Business logic is business logic.
Steve: And that stuff, that's all created runtime, right? So you don't need to rebuild the model for that either?
Nico Acosta: Yeah, no, you don't need to rebuild the model. That is completely created at runtime.
Steve: Yeah, so you could imagine this, that the questions are being asked being truly personal and maybe generated from backend data in some cases.
Nico Acosta: Absolutely, absolutely.
Steve: Very cool.
Nico Acosta: And the default behaviors here, you essentially tell the assistant where to start. You tell the assistant which task you want to use as a fallback, in case it doesn't understand. And you give it a, you point this to the same fallback. On failure, if it fails more than three times to collect the question, it goes to fallback.
Nico Acosta: And the stylesheet is really a unique concept that we have, which gives the tone and a voice to your assistant. So here, we set the voice, and we set the error messages, or like, the default error messages. You can customize these per question if you want, but we take care of all the validation, so it's not something that you need to worry about. You just customize the messages, and that gives you kind of like the tone, and voice, and the feel for the assistant.
Steve: Yeah, so you can change all this out without having to rebuild the assistant, right? And so, maybe even this potentially could be personalized based on the user persona that's coming in. Maybe if you had like an older user, a user from a certain region, versus a younger user, you might change the voice out. And is that using Amazon Polly Voices?
Nico Acosta: Yeah, that's using Amazon Polly Voices, and we're constantly adding more voices, and we want to add like a whole set of voices, so you have pretty much access to whatever is out there. So, like, the voices are, everybody wants a different voice, wants their own voice. So, here, it's a place where we believe in selection. You just need to have access to a ton of voices.
Steve: Is there like the Twilio Nico voice yet? I don't see that one.
Nico Acosta: Not yet.
Steve: Down on the roadmap, right?
Nico Acosta: So, that's a quick overview. Let's see what else. And then, there's the channels. So, here-
Steve: Yeah, this is my favorite part.
Nico Acosta: Essentially, you can deploy the assistant to wherever you want. If you're not familiar with Twilio, this could be a little bit strange that you just get a URL, but it all makes sense when you go to a phone number configuration, say, insurance. And then, the way that you configure a phone number is with a URL. So, essentially, when a call comes in, webhook, and we tell it webhook to Autopilot. So, we copy this URL and this phone number for voice, and then we do the same for messaging.
Nico Acosta: So, you can actually just give this phone number a call. 424-276-6333.
Speaker 3: Welcome to the Conversational Insurance Company. I'm your virtual assistant. How can I help you today?
Nico Acosta: I want to know the status of my claim.
Speaker 3: You have one active claim number, 234237, for your 2007 Lamborghini Diablo. It has been accepted and is pending payment. Is there anything else I can help you with?
Nico Acosta: So, you can see how it's really easy to get it connected to voice.
Steve: And now we know they do cover Lamborghinis.
Nico Acosta: They do cover Lamborghinis. So, here we can see how, we saw how you can configure Autopilot to work with a phone number. And one of the cool things and something I haven't showed you, Steve, is how it works with chat and Flex.
Nico Acosta: So, think, we have not changed one line of code. You just saw it working over the phone. If you text the number, if you text this number, it'll work the same way.
Steve: Oh, there's a web chat.
Nico Acosta: Imagine this is the website for the Conversational Insurance Company, and say, "Hi there." Welcome to Conversational Insurance Company. I'm your virtual assistant. How can I help you? And I want to know what is the status of my claim. And then, you get, you have one active claim, number 223, for your 2007 Lamborghini Diablo. It's pending payment. But I imagine I really need that paid really fast, so it's like, can I talk to an agent? And notice here, I'm Maria, this is Autopilot.
Nico Acosta: Okay, I'll connect you to an agent now, please hold on. And what we'll see here in Twilio Flex, this is our contact center application platform, and I'm logged in as an agent as well. You'll see a task come in. So, you have Maria, which I'm Maria on this tab, and I can see an incoming chat request. And this goes through a whole routing layer that, obviously, I'm the only agent in this install, but if you have a set of agents, it goes through a whole routing layer that does like skill based routing or any rule based routing that you want, until you get past here.
Nico Acosta: And one thing that you'll see, which is really cool, is that the agent gets the transcript of what the user interacted with the bot.
Steve: Yeah, that's so awesome.
Nico Acosta: So, like, the user, you already get all the context here. You're seeing like, okay, you check the claim, and then, you're going to respond, "Hi Maria, this is Nico. How can I help you?" And then, you'll go back to Maria's window, and you'll see kind of where Autopilot left off, and then you have agent Nico post that Maria, how can I help you. And then you, can I get paid sooner? Sure. Let me sort that out for you.
Steve: Yeah, this is very cool.
Nico Acosta: And you have here eventually, the agent, the interaction is done. And here, to the right pane, you can configure a whole CRM and do a whole bunch of stuff. So, like, that smooth hand off is really important, because we believe that there are some tasks that are really well suited for self service, like getting an insurance quote, and that can get complicated, but some tasks have very defined workflows that are well suited for self service. Other tasks are suited for like a hybrid. You want to do a part with a bot, a part with an agent.
Nico Acosta: And then, other tasks, frankly, you kind of do want to route to an agent directly. Something like file a claim, or say, report an accident, right? If you say, "Hey, I'm going to report an accident," you want to be tied to a human. You want to show some, like, you want to get like advice, and it's a tricky situation, right, if you're involved in a car crash. So, you don't want to be-
Steve: Somebody's all emotional. Yeah, they've just gotten in an accident. You don't want them talking to a bot.
Nico Acosta: Yeah, so, that's kind of what the combo of Autopilot and Flex does, where Autopilot can do the task recognition, detect that you're filing a claim. If you look at the samples here, we have some that are report an accident, I got into a car crash, right? I crashed my car, car crash, right? These need escalation immediately, and with Flex, once we hand off to Flex, you have like different cues with different priorities.
Nico Acosta: So like, the person that escalates on, like the standard escalation gets one priority, and the person that's filing a claim gets a different priority when he hands off to the agent. So, it's kind of like this interplay between automation and leveraging our human agents that we believe is so powerful. So here, as an agent, you're going to complete and say, okay, nope, this is completed. And then, no active tasks. On here, I say okay, I'm done for the day, I'm offline. You can see here how, thanks for chatting with us, I can start another chat.
Nico Acosta: So, that's kind of like a little bit the new stuff that I'm showing you. I'm super, super excited about that I hadn't showed you before, and we believe it's going to be really, really powerful. Normally, these are like two separate worlds, where like, you have to stitch them together, or like, you have to ... And when you deploy in production, when you deploy in the real world, the reality is that you need these two worlds to work in consonance, and we believe that that is key to the success, kind of.
Nico Acosta: Autopilot's not, we don't assume that we will build a platform that will past the Turing test at the first go, and we realize the reality that working together with the contact center agent is the best possible customer experience. So, and we put a lot of effort into that smooth hand off, and you can do the same in voice, which is also pretty cool.
Steve: Yeah, I think that's the reality, where this stuff is really effective, it's not humans working bots, or bots working without humans, but the combination of the two, where the bots can do things that really free humans up to do human kinds of tasks, and maybe that's, in the case of a car accident, just providing empathy that a bot is not going to be able to do, or answering complex questions that it just doesn't make sense to try to get a bot to do. And I love this, the seamless hand off. I was-
Nico Acosta: Go ahead.
Steve: No, I was just going to say, we were working on a few bigger projects using Autopilot, and one client, we were working on an Alexa skill backed by Autopilot, and he called and was talking to me. He said, "Well, out now, I don't have access to an Echo, so I'll try it when I get home." I said, "Hold on," and I set up a telephone number, and within a couple of minutes, I said, "Hey, just call this number and you can test it that way." And he was just blown away. He's like, "I thought we were building an Alexa skill." I said, "Well, we are, but the way that we're building it really allows users to access it from lots of different channels."
Steve: And I think as developers, we need to be thinking that way more. Like, it isn't going to be, you know, we're building once for Alexa, or for Google, or for SMS, or for, you know, pick your channel. Users are going to pick the channel that they want to use, and beyond that, they're going to very likely change from channel to channel. Like, you have somebody that today will ask a question, because they're at home, through an Echo, and then they'll be in their car, and well, hopefully they aren't texting while they're driving. They'll be out of the house and they'll want to access via text message or via telephone. And the assistants should be accessible anywhere and through any means that the user chooses.
Nico Acosta: Yeah, one of the things that we think a lot about is kind of the importance of separating the intelligence layer from the channel. When you build your models and all your intelligence layer on a channel-specific tool, you can't leverage that across. And the models become more robust when they see more types of examples, so one of the things you'll notice here is that we have like all the different ways that you say stuff, and here is like, you notice a typo, hi there, still got that, that it was greeting.
Nico Acosta: But here, you see, for example, the one that came through voice, right? And then, one cool thing is that with this, you have a supervised learning loop, where you can take these queries and create a sample from them. So, can I talk to an agent? That is talk to representative, and you add it, and the next time you build the model, and it even kind of keeps track of which ones you've reviewed and which ones you're pending review. So like, you could say, okay, this one, discard it, this one, discard it. Only show me the ones that are pending review. And then it helps you kind of keep track all of that supervised learning loop.
Nico Acosta: So that you can keep all this stuff, if you train it with samples that come from messaging, that come from voice, that come from Alexa, it'll build a much more robust model, and then you'll be able to use it across, than kind of building in different models for each of the platforms. If kind of fragmenting code is bad, fragmenting the models is kind of 10 times bad.
Steve: Yeah, yeah, or building seven different versions of a bot so that you can support different channels. I know having that information, what channel it's coming in, on the backend code responding is important to, because the way that you would respond, or the messaging that you would use, say if you were responding to somebody that is coming in through web chat might be very different than how you would respond if it's an audio response.
Nico Acosta: Yeah, and that's the key of being able to have these kind of dynamic actions, where here, we pass, always you have like event.CurrentTask. But if you do like event.Channel, we pass the channel, and then you can customize the message for each channel if you want, so that you have kind of a unique experience.
Nico Acosta: And a couple of cool things that we announced recently, or a really cool thing is the WhatsApp integration. So, we have Facebook Messenger, we have Slack, but WhatsApp, you're able to build a WhatsApp bot with like the same characteristics, like being able to hand off to an agent, doing all the same things, but instead of being here, it's on a WhatsApp number, or yeah, on WhatsApp. So, it's pretty cool.
Nico Acosta: And like, you got it working for web chat, you got it working for SMS, you got it working for Alexa, for voice, then you type in WhatsApp, you kind of get that for absolutely free. And then, there's really interesting stuff coming. All the stuff that's happening with RCS and like the Android ecosystem on messaging, that's really interesting. I know where Apple Business Chat is going is really interesting as well.
Steve: And on the API, they're using like just a generic channel. You really aren't even limited to those channels there. Those are the most popular and most obvious, but you could really, through the API, plug into any channel.
Nico Acosta: Yeah, you could build your, integrate your own channel. If you have, for example, your custom build web chat, you can absolutely, we have an endpoint for custom channel, and you can absolutely hit that for your own web chat, or if you want to integrate with a channel that we don't support, so you're not trapped into like, waiting for the roadmap or anything. You can build it yourself.
Steve: Yeah, I know. We've been super excited about it, and having a lot of fun working on the projects that we're working on with you guys, and with clients of ours that are now backed by Autopilot, and it's come a long way in a short period of time. It's come a long, long way. I know that didn't happen by magic. I know you guys are doing a lot of work. But I think maybe the first time I saw it was up in New York in what, June of last year, and it doesn't, I mean, yeah, it's a completely different product in less than a year.
Twilio Developer Community Hackathons
Nico Acosta: Yeah, but it's also cool to talk a little bit about those hackathons on how we built this product. And we really built this product with the developer community. We did, normally how you build a product. Like, you're going to research the stuff. You build, build, build, build, and then you do some sort of beta where you invite customers.
Nico Acosta: We took a different approach with Autopilot. We did a series of hackathons. Throughout the world, we did four hackathons. And the first version of the hackathon was like something that, looking back, it was irresponsible to put in front of the customers. It was like, it was barely working. You saw the second version. Then, from New York, we went to London, then we came back to New York, and-
Steve: Yeah, I saw the second version. Was that the version?
Nico Acosta: Yeah, yeah. And that's a really cool way to build a product, build it with the community, build it with folks that have great ideas on what they want to build, and they just come to the hackathons and go at it, and you see what's working and what's not, and you're able to iterate really quickly. The other thing is that we take the full team to the hackathon, all the engineers, the data scientists, the ML engineers. We take them to the hackathon, and they're working with customers, and everybody's helping customers. So, we are a very customer-focused company, and those hackathons were kind of a really cool way to build this product.
Steve: Yeah, I really enjoyed it, and I got to say, when I went, I kind of expected to see a lot of the same, you know, because there's, I know there's dialog workflows in Amazon Lex, and Google, or in Alexa course, and some other ones out there. So, I kind of expected that, but was really pleasantly surprised, and mostly in understanding early on the vision for building bots that could work with humans. I think that is just, makes a ton of sense, especially in business use cases.
Steve: And in one platform, to build bots that will work across whatever channel the user selects, I think that is, you know, we can't tell people to use one channel. They're going to pick whatever channel they want to use. And so, you know.
Nico Acosta: And we don't know what's coming. We don't know what's coming or if it's going to be a new messaging ecosystem that's going to take over, or maybe it's going to take over in a part of the world, or a part, like for a human demographic. Like, we don't know, and kind of betting on an approach that is future proof is really important.
Steve: Yeah, no, I went to an event not too long ago that was actually like a VR, you know, virtual reality, and they've got digital assistants that were working on the company that I was meeting with, and doing a lot of cool things. And all of a sudden, you start seeing where those brand new messaging channels are going to come up for gameplay or for VR worlds and things like that, and being able to tie into, again, whatever channel the user is. Wherever they are is where you got to be.
Nico Acosta: Yeah, yeah, yeah.
Steve: Well, this was really great. I really appreciate you taking the time out. I know you just have a ton of plates in the air, and finding extra time is not easy, so I appreciate that. And if somebody has questions, where should they go? What should they do to contact you or contact the team?
Nico Acosta: Yeah, so, check out Twilio.com/autopilot, or Twilio.com/docs/autopilot. There you'll find all the documentation, so, you'll find all the documentation on how to get it started. You'll find a couple of tutorials that are really kind of key to follow. So, here on docs/autopilot, you'll get a bunch of videos and content on how to get started.
Nico Acosta: If you need to reach out to us, just shoot us an email to help@twilio. The product is in beta, and we are watching all the tickets very closely, and we're happy to chat with you guys and help you get started, and hear what you're looking to build.
Steve: Awesome. Well, hey, again, I'll say thank you, and I'll let you get back to it, and maybe we can regroup in a few months and see what's new, if it's [crosstalk 00:40:11]-
Nico Acosta: Absolutely.
Steve: It might be a whole new world.
Nico Acosta: Yeah, absolutely, and thank you, Steve, and thank you for the invite.
Steve: All right. We'll see you.
Nico Acosta: Okay, bye bye.
This transcript and video are posted with permission from Dabble Lab.
Hope you enjoyed this transcript of Steve and Nico's session. Be sure to check out the documentation for getting started with Twilio Autopilot and apply to become a Twilio Champion today.
Wanna talk more about bots? Read our guide for bot use cases and best practices.