Menu

Expand
Rate this page:

Automated Survey with Node.js and Express

Twilio is launching a new Console. Some screenshots on this page may show the Legacy Console and therefore may no longer be accurate. We are working to update all screenshots to reflect the new Console experience. Learn more about the new Console.

Implementing your own automated survey with Twilio Voice and SMS can be a huge time saver when you need to collect feedback from a group of people. Whether it's participants in a social services program or a field service organization, you can quickly set up your own survey to collect structured data over the phone or via text message. Here's how it works at a high level:

  1. The end user calls or texts the survey phone number.
  2. Twilio gets the call or text and makes an HTTP request to your application for instructions on how to respond.
  3. Your web application serves up TwiML instructions to Gather or Record the user input over the phone, or prompts for text input with Message.
  4. After each question, Twilio makes another request to your server with the user's input, which your application stores in its database.
  5. Your application returns a TwiML response to Twilio with instructions to either ask the next question or end the survey.

What We Will Learn

This How-To demonstrates how to use TwiML to deliver a survey that can be completed via voice call. The survey actually works via SMS text messages, too, but we're going to focus on the looping logic necessary to conduct an interview over the phone. We will create a voice call flow using the Say, Record and Gather TwiML verbs.

You will also learn how to maintain conversation state in a database that spans multiple webhook requests. Beyond an automated survey, these techniques can be applied to implement more complex IVR systems or text message interfaces.

Instacart uses Twilio to power their customer service surveys and integrate that feedback into their customer database. Read more here.

Loading Code Sample...
        
        
        package.json

        Specify your application dependencies (configuration file)

        package.json
        Click here to get started!

        About This Application

        Like most Node.js web applications, this one relies on a number of smaller modules installed via npm to handle HTTP requests and store data. The key modules for this application are:

        • express - a popular web framework that helps us respond to HTTP requests to our application.
        • mongoose - an Object/Document Mapper (ODM) for MongoDB.
        • twilio - the Twilio Node module will help us generate TwiML responses to drive our interview.
        Loading Code Sample...
              
              
              package.json

              Include application dependencies

              package.json
              Bootstrap the application

              Bootstrapping the Application

              This is the Node file we will execute to serve up our web application. We load our app's configuration from an external file containing the HTTP port we want to run on and the MongoDB database connection string we need to store data using Mongoose.

              We also define four routes to be handled by our web application. Three of the routes are webhooks that will be requested by Twilio when your Twilio survey number receives an incoming call or text, or when the results of a transcription job are ready. The fourth route will be used by our reporting UI to get the results of the survey from the database.

              Loading Code Sample...
                    
                    
                    index.js

                    Load application routes and bind a server to the specified port

                    index.js

                    Now that our application is all set up, let's look at the high level steps necessary to implement a voice interview via Twilio and TwiML.

                    What is an interview loop?

                    The Voice Interview Loop

                    The user can enter input for your survey over the phone using either their phone's keypad or by speaking. After each interaction Twilio will make an HTTP request to your web application with either the string of keys the user pressed or a URL to a recording of their voice input.

                    Loading Code Sample...
                          
                          
                          routes/voice.js

                          Handle the main interview loop for phone calls

                          routes/voice.js

                          It's up to our application to process and store the user's input, maintain the current state of the conversation, and respond back to the user. Let's dive into this flow to see how it actually works.

                          Configure your application to work with Twilio

                          Responding to a Phone Call

                          To initiate the interview process, we need to configure one of our Twilio numbers to send our web application an HTTP request when we get an incoming call or text (remember, our app accepts text messages as well, even though we're focusing on the voice side).

                          Click on one of your numbers and configure Voice and Message URLs that point to your server. In our code, the routes are /voice and /message, respectively.

                          Automated Survey Webhook Setup

                          If you don't already have a server configured to use as your webhook, ngrok is a great tool for testing webhooks locally.

                          We've configured our webhooks in the Twilio Console. Next let's see how to respond to requests.

                          Respond to a voice call

                          Responding to a Phone Call

                          The voice route maps to this Express handler function, which takes an HTTP request and HTTP response as arguments. From the request, we can access the phone number of the person calling in by the From POST parameter - we can use this to uniquely identify a person taking the survey.

                          We can also access the RecordingUrl, which contains any voice input from the user. Digits may also be present, which contains the string of keys entered by the user on their keypad. If this is the user's first call to our system or they failed to enter any input to the previous question, these values might be blank Strings.

                          We also create a TwimlResponse object that we will use to build up a string of XML we can ultimately render as a response to Twilio's request. It's not doing anything fancy - it just provides a JavaScript object that we can use to progressively assemble a valid TwiML string as our program executes.

                          Loading Code Sample...
                                
                                
                                routes/voice.js

                                Get phone information and input from the request object

                                routes/voice.js

                                We've seen how to handle requests to our webhooks. Now lets go deeper into how to generate TwiML to redirect our users to the next question and generate speech from text.

                                Ask a question!

                                Asking a Question

                                If either the user did not enter any input, it's the first question in the survey, or there's still another question after the current one, we will build a TwiML response that will ask the next question.

                                We define a few inner functions here to help us build our response. respond completes our TwiML response and sends XML content back to Twilio. say is shorthand for appending a string of text that will be read back to the user with Twilio TTS (text-to-speech) engine.

                                Loading Code Sample...
                                      
                                      
                                      routes/voice.js

                                      Build voice responses with TwiML

                                      routes/voice.js
                                      Gather user responses

                                      Asking a Question

                                      In our TwiML response, we need to include either a Gather tag or a Record tag to collect input from the user. Which tag we use depends on the question type in the survey.

                                      In the Record use case, we also provide a transcription callback URL. Unlike Gather and Record, the transcription callback happens outside the loop of the call, sometime in the very near future (several seconds rather than minutes). So while you can't count on having the transcript results during the flow of the call, you can add the transcript to your database record to give the response a chance to help enrich that data a bit.

                                      Another caveat here is that Twilio's transcription service is automated and not always super accurate. If transcription accuracy is critical for you, you might consider using a service with human translators like Rev.com.

                                      Loading Code Sample...
                                            
                                            
                                            routes/voice.js

                                            Gather user input for various question types

                                            routes/voice.js

                                            Now that we know how to ask questions and gather user input. Lets see how to store the survey state.

                                            Where are we in this conversation?

                                            Updating Conversation State

                                            Saving the user's response and maintaining the state of our conversation with the user is a concern best handled at the model layer of our server-side application, so we use our MongoDB-backed Mongoose model to handle this for us.

                                            Abstracting the survey state from the controller also has the benefit of letting us re-use it for handling survey inputs from text messages. #winning!

                                            Loading Code Sample...
                                                  
                                                  
                                                  routes/voice.js

                                                  Route to an existing survey or create a new one

                                                  routes/voice.js

                                                  We have a general idea of how we want to persist survey state. Lets go into more details and have a look at what our schema looks like.

                                                  The Mongoose schema

                                                  The Mongoose Schema

                                                  In our model, we create a schema that allows us to constrain and validate the form of documents that we insert into MongoDB. However, the Mixed type lets us store arbitrary JavaScript objects in the responses array, so we have some flexibility there with the objects we use to store our user's answers.

                                                  Loading Code Sample...
                                                        
                                                        
                                                        models/SurveyResponse.js

                                                        Define survey response model schema

                                                        models/SurveyResponse.js

                                                        Next, lets see how we will keep track of our data by using the type property on our models.

                                                        Store survey responses

                                                        Storing Responses

                                                        Prior to saving a response from the user, we convert the raw string values submitted into data types that will be easier for us to work with as we analyze and visualize our survey data.

                                                        Loading Code Sample...
                                                              
                                                              
                                                              models/SurveyResponse.js

                                                              Convert and save user responses

                                                              models/SurveyResponse.js

                                                              We've seen how to persist answers to our questions. Next we'll see how guide the user to the next question.

                                                              Step through the survey flow

                                                              Queueing Up the Next Question

                                                              After the current response is saved, we invoke the callback to our controller with the index of the next question in the survey. This may be longer than the actual length of the survey, which means we're finished asking questions!

                                                              Loading Code Sample...
                                                                    
                                                                    
                                                                    models/SurveyResponse.js

                                                                    Save response and mark survey as done if there are no more questions

                                                                    models/SurveyResponse.js

                                                                    After you've finished asking questions, you might be interested in seeing how people responded to your survey. We won't spend too much time on visualizing the data, but we've included a bit of code in this sample app that shows you how you might approach that.

                                                                    Display survey results

                                                                    Checking Out The Results

                                                                    In this app's static asset directory, you'll find an index.html file that contains some markup for displaying the results of our survey questions. It makes an Ajax request to our Express application to get the last 100 results from our survey to display here.

                                                                    For phone survey responses, there is a link to listen to the recording in addition to the transcribed text sent to our application.

                                                                    Loading Code Sample...
                                                                          
                                                                          
                                                                          public/index.html

                                                                          Render results for your survey with HTML

                                                                          public/index.html

                                                                          Sometimes, the results aren't enough. So lets see how we can drill deeper and hear the actual recordings.

                                                                          Listen to survey recordings

                                                                          Playing Twilio Recordings

                                                                          In the JavaScript that dynamically inserts the recording links into the page, we open the same RecordingUrl sent to us from Twilio in a new window. This works well enough, but you might consider playing them directly on the page with a more robust audio library like Buzz. When preparing the recordings to be played via HTML 5 audio, note that each recording is made available in either WAV or MP3 formats.

                                                                          Loading Code Sample...
                                                                                
                                                                                
                                                                                public/index.js

                                                                                Load survey responses from the server

                                                                                public/index.js
                                                                                That's all, folks!

                                                                                That's All Folks

                                                                                And that's it! You can reload the page on your localhost in your browser and watch as the results fly in from your users.

                                                                                Loading Code Sample...
                                                                                      
                                                                                      
                                                                                      public/index.js

                                                                                      Render survey responses as charts

                                                                                      public/index.js
                                                                                      What's next?

                                                                                      Where to next?

                                                                                      If you're a Node.js developer working with Twilio, you might also enjoy these tutorials:

                                                                                      Click To Call

                                                                                      Learn how to use Twilio Client to convert web traffic into phone calls with the click of a button.

                                                                                      Two Factor Authentication

                                                                                      Learn to implement two-factor authentication (2FA) in your web app with Twilio-powered Authy.

                                                                                      Did this help?

                                                                                      Thanks for checking this tutorial out! If you have any feedback to share with us, we'd love to hear it. Connect with us on Twitter and let us know what you build!

                                                                                      Kevin Whinnery Orlando Hidalgo Kat King Mica Swyers Andrew Baker Jose Oliveros Paul Kamp Brianna DelValle
                                                                                      Rate this page:

                                                                                      Need some help?

                                                                                      We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd by visiting Twilio's Stack Overflow Collective or browsing the Twilio tag on Stack Overflow.

                                                                                      Loading Code Sample...
                                                                                            
                                                                                            
                                                                                            

                                                                                            Thank you for your feedback!

                                                                                            Please select the reason(s) for your feedback. The additional information you provide helps us improve our documentation:

                                                                                            Sending your feedback...
                                                                                            🎉 Thank you for your feedback!
                                                                                            Something went wrong. Please try again.

                                                                                            Thanks for your feedback!

                                                                                            thanks-feedback-gif