Automated Survey with Node.js and Express

Implementing your own automated survey with Twilio Voice and SMS can be a huge time saver when you need to collect feedback from a group of people. Whether it's participants in a social services program or a field service organization, you can quickly set up your own survey to collect structured data over the phone or via text message. Here's how it works at a high level:

1. The end user calls or texts the survey phone number.
2. Twilio gets the call or text and makes an HTTP request to your application for instructions on how to respond.
3. Your web application serves up TwiML instructions to Gather or Record the user input over the phone, or prompts for text input with Message.
4. After each question, Twilio makes another request to your server with the user's input, which your application stores in its database.
5. Your application returns a TwiML response to Twilio with instructions to either ask the next question or end the survey.

What We Will Learn

This How-To demonstrates how to use TwiML to deliver a survey that can be completed via voice call. The survey actually works via SMS text messages, too, but we're going to focus on the looping logic necessary to conduct an interview over the phone. We will create a voice call flow using the Say, Record and Gather TwiML verbs.

You will also learn how to maintain conversation state in a database that spans multiple webhook requests. Beyond an automated survey, these techniques can be applied to implement more complex IVR systems or text message interfaces.

Instacart uses Twilio to power their customer service surveys and integrate that feedback into their customer database. Read more here.

About This Application

Like most Node.js web applications, this one relies on a number of smaller modules installed via npm to handle HTTP requests and store data. The key modules for this application are:

• express - a popular web framework that helps us respond to HTTP requests to our application.
• mongoose - an Object/Document Mapper (ODM) for MongoDB.
• twilio - the Twilio Node module will help us generate TwiML responses to drive our interview.

Bootstrapping the Application

This is the Node file we will execute to serve up our web application. We load our app's configuration from an external file containing the HTTP port we want to run on and the MongoDB database connection string we need to store data using Mongoose.

We also define four routes to be handled by our web application. Three of the routes are webhooks that will be requested by Twilio when your Twilio survey number receives an incoming call or text, or when the results of a transcription job are ready. The fourth route will be used by our reporting UI to get the results of the survey from the database.

Now that our application is all set up, let's look at the high level steps necessary to implement a voice interview via Twilio and TwiML.

The Voice Interview Loop

The user can enter input for your survey over the phone using either their phone's keypad or by speaking. After each interaction Twilio will make an HTTP request to your web application with either the string of keys the user pressed or a URL to a recording of their voice input.

It's up to our application to process and store the user's input, maintain the current state of the conversation, and respond back to the user. Let's dive into this flow to see how it actually works.

Responding to a Phone Call

To initiate the interview process, we need to configure one of our Twilio numbers to send our web application an HTTP request when we get an incoming call or text (remember, our app accepts text messages as well, even though we're focusing on the voice side).

Click on one of your numbers and configure Voice and Message URLs that point to your server. In our code, the routes are /voice and /message, respectively.

If you don't already have a server configured to use as your webhook, ngrok is a great tool for testing webhooks locally.

We've configured our webhooks in the Twilio Console. Next let's see how to respond to requests.

Responding to a Phone Call

The voice route maps to this Express handler function, which takes an HTTP request and HTTP response as arguments. From the request, we can access the phone number of the person calling in by the From POST parameter - we can use this to uniquely identify a person taking the survey.

We can also access the RecordingUrl, which contains any voice input from the user. Digits may also be present, which contains the string of keys entered by the user on their keypad. If this is the user's first call to our system or they failed to enter any input to the previous question, these values might be blank Strings.

We also create a TwimlResponse object that we will use to build up a string of XML we can ultimately render as a response to Twilio's request. It's not doing anything fancy - it just provides a JavaScript object that we can use to progressively assemble a valid TwiML string as our program executes.

We've seen how to handle requests to our webhooks. Now lets go deeper into how to generate TwiML to redirect our users to the next question and generate speech from text.

Asking a Question

If either the user did not enter any input, it's the first question in the survey, or there's still another question after the current one, we will build a TwiML response that will ask the next question.

We define a few inner functions here to help us build our response. respond completes our TwiML response and sends XML content back to Twilio. say is shorthand for appending a string of text that will be read back to the user with Twilio TTS (text-to-speech) engine.

Asking a Question

In our TwiML response, we need to include either a Gather tag or a Record tag to collect input from the user. Which tag we use depends on the question type in the survey.

In the Record use case, we also provide a transcription callback URL. Unlike Gather and Record, the transcription callback happens outside the loop of the call, sometime in the very near future (several seconds rather than minutes). So while you can't count on having the transcript results during the flow of the call, you can add the transcript to your database record to give the response a chance to help enrich that data a bit.

Another caveat here is that Twilio's transcription service is automated and not always super accurate. If transcription accuracy is critical for you, you might consider using a service with human translators like Rev.com.

Now that we know how to ask questions and gather user input. Lets see how to store the survey state.

Updating Conversation State

Saving the user's response and maintaining the state of our conversation with the user is a concern best handled at the model layer of our server-side application, so we use our MongoDB-backed Mongoose model to handle this for us.

Abstracting the survey state from the controller also has the benefit of letting us re-use it for handling survey inputs from text messages. #winning!

We have a general idea of how we want to persist survey state. Lets go into more details and have a look at what our schema looks like.

The Mongoose Schema

In our model, we create a schema that allows us to constrain and validate the form of documents that we insert into MongoDB. However, the Mixed type lets us store arbitrary JavaScript objects in the responses array, so we have some flexibility there with the objects we use to store our user's answers.

Next, lets see how we will keep track of our data by using the type property on our models.

Storing Responses

Prior to saving a response from the user, we convert the raw string values submitted into data types that will be easier for us to work with as we analyze and visualize our survey data.

We've seen how to persist answers to our questions. Next we'll see how guide the user to the next question.

Queueing Up the Next Question

After the current response is saved, we invoke the callback to our controller with the index of the next question in the survey. This may be longer than the actual length of the survey, which means we're finished asking questions!

After you've finished asking questions, you might be interested in seeing how people responded to your survey. We won't spend too much time on visualizing the data, but we've included a bit of code in this sample app that shows you how you might approach that.

Checking Out The Results

In this app's static asset directory, you'll find an index.html file that contains some markup for displaying the results of our survey questions. It makes an Ajax request to our Express application to get the last 100 results from our survey to display here.

For phone survey responses, there is a link to listen to the recording in addition to the transcribed text sent to our application.

Sometimes, the results aren't enough. So lets see how we can drill deeper and hear the actual recordings.

Playing Twilio Recordings

In the JavaScript that dynamically inserts the recording links into the page, we open the same RecordingUrl sent to us from Twilio in a new window. This works well enough, but you might consider playing them directly on the page with a more robust audio library like Buzz. When preparing the recordings to be played via HTML 5 audio, note that each recording is made available in either WAV or MP3 formats.

That's All Folks

And that's it! You can reload the page on your localhost in your browser and watch as the results fly in from your users.

Where to next?

If you're a Node.js developer working with Twilio, you might also enjoy these tutorials:

Click To Call

Learn how to use Twilio Client to convert web traffic into phone calls with the click of a button.

Two Factor Authentication

Learn to implement two-factor authentication (2FA) in your web app with Twilio-powered Authy.

Did this help?

Thanks for checking this tutorial out! If you have any feedback to share with us, we'd love to hear it. Connect with us on Twitter and let us know what you build!