Automated Survey with Java and Spark

Time to read: 3 minutes

January 10, 2017
Written by
Jose Oliveros
Contributor
Opinions expressed by Twilio contributors are their own
Reviewed by
Paul Kamp
Twilion
Samuel Mendes
Contributor
Opinions expressed by Twilio contributors are their own
Andrew T. Baker
Twilion
Kat King
Twilion
Orlando Hidalgo
Contributor
Opinions expressed by Twilio contributors are their own
Brianna DelValle
Twilion
Jeffrey Linwood
Contributor
Opinions expressed by Twilio contributors are their own

automated-survey-java-spark

This Spark application uses the Twilio API to create an automated survey conducted over SMS or phone call. Callers will interact with this application over the phone, and you will be able to view the results on a dynamic dashboard.

Check out the source code for this application on GitHub.

Creating a Survey

Before we gather responses, let's specify the questions we would like to ask.

Each question is created as a JSON object with two attributes: text, which holds the question we'd like to ask, and type, which describes the kind of input we expect a caller to provide when asked that question.

Our app reads this JSON into a Java object using Google's Gson library.

In this tutorial, we'll highlight the code that interacts with Twilio and in turn makes the application tick. Check out the project README on GitHub to see how to run the code yourself.

Editor: this is a migrated tutorial. Find the original code at https://github.com/TwilioDevEd/automated-survey-spark/


    {
        text: 'Please tell us your age.'

Our users will need to contact us in order to take our survey. Let's see how to accept a call or SMS.

Accepting a Call

Every time your Twilio number receives a call or an SMS, Twilio will send an HTTP request to the application asking what to do next. The application responds with TwiML that describes the action (<Say> a phrase, <Gather> input, <Message> an SMS, among others).

Your Twilio number must be configured to make requests to the /interview route of this application. Go to your Manage Numbers page, and set up a new TwiML application using one of your Twilio numbers.

If you don't already have a server configured to use as your webhook, ngrok is a great tool for testing webhooks locally.

package com.twilio.survey.controllers;

import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.util.HashMap;
import java.util.Map;

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import com.twilio.survey.Server;
import com.twilio.survey.models.Response;
import com.twilio.survey.models.Survey;
import com.twilio.survey.models.SurveyService;
import com.twilio.survey.util.IncomingCall;

import spark.Route;

public class SurveyController {
  private static SurveyService surveys = new SurveyService(Server.config.getMongoURI());

  // Main interview loop.
  public static Route interview = (request, response) -> {
    Map<String, String> parameters = parseBody(request.body());
    IncomingCall call = IncomingCall.createInstance(parameters);
    AbstractMessageFactory messageFactory = AbstractMessageFactory.createInstance(parameters);

    Survey existingSurvey = surveys.getSurvey(call.getFrom());
    if (existingSurvey == null) {
      Survey survey = surveys.createSurvey(call.getFrom());
      return messageFactory.firstTwiMLQuestion(survey);
    } else if (!existingSurvey.isDone()) {
      existingSurvey.appendResponse(new Response(call.getInput()));
      surveys.updateSurvey(existingSurvey);
      if (!existingSurvey.isDone()) {
        return messageFactory.nextTwiMLQuestion(existingSurvey);
      }
    }
    return messageFactory.goodByeTwiMLMessage();
  };


  // Results accessor route
  public static Route results = (request, response) -> {
    Gson gson = new Gson();
    JsonObject json = new JsonObject();
    // Add questions to the JSON response object
    json.add("survey", gson.toJsonTree(Server.config.getQuestions()));
    // Add user responses to the JSON response object
    json.add("results", gson.toJsonTree(surveys.findAllFinishedSurveys()));
    response.type("application/json");
    return json;
  };

  // Transcription route (called by Twilio's callback, once transcription is complete)
  public static Route transcribe = (request, response) -> {
    IncomingCall call = IncomingCall.createInstance(parseBody(request.body()));
    // Get the phone and question numbers from the URL parameters provided by the "Record" verb
    String surveyId = request.params(":phone");
    int questionId = Integer.parseInt(request.params(":question"));
    // Find the survey in the DB...
    Survey survey = surveys.getSurvey(surveyId);
    // ...and update it with our transcription text.
    survey.getResponses()[questionId].setAnswer(call.getTranscriptionText());
    surveys.updateSurvey(survey);
    response.status(200);
    return "OK";
  };

  // Helper methods

  // Spark has no built-in body parser, so let's roll our own.
  public static Map<String, String> parseBody(String body) throws UnsupportedEncodingException {
    String[] unparsedParams = body.split("&");
    Map<String, String> parsedParams = new HashMap<String, String>();
    for (int i = 0; i < unparsedParams.length; i++) {
      String[] param = unparsedParams[i].split("=");
      if (param.length == 2) {
        parsedParams.put(urlDecode(param[0]), urlDecode(param[1]));
      } else if (param.length == 1) {
        parsedParams.put(urlDecode(param[0]), "");
      }
    }
    return parsedParams;
  }

  public static String urlDecode(String s) throws UnsupportedEncodingException {
    return URLDecoder.decode(s, "utf-8");
  }
}

Now that we can receive a call, let's look at how to ask our user a question.

Asking a Question

Once the application receives an HTTP request from Twilio, it uses a different factory based on whether an SMS or phone call generated the request. This factory helps us build the appropriate TwiML response.

For SMS requests, the application will use the <Message> verb to answer all requests.

For requests generated by phone calls, the TwiML will use the <Say> verb for the question's text, and <Gather> or <Record> to collect input based on expected user input.

TwiML verbs have properties that allow you to change the way the verb works. For example, the <Record> verb always collects audio input, but using the transcribeCallback property, you can specify a route to which Twilio will send a transcript of the audio input. For more on TwiML verbs and their properties, check out the TwiML API documentation.

package com.twilio.survey.controllers;

import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;

import com.twilio.survey.Server;
import com.twilio.survey.models.Survey;
import com.twilio.survey.util.Question;

import com.twilio.twiml.VoiceResponse;
import com.twilio.twiml.voice.Gather;
import com.twilio.twiml.voice.Record;
import com.twilio.twiml.voice.Say;
import com.twilio.twiml.TwiMLException;


public class TwiMLMessageFactory extends AbstractMessageFactory {

  String firstTwiMLQuestion(Survey survey) throws TwiMLException, UnsupportedEncodingException {
    VoiceResponse.Builder voiceResponseBuilder = new VoiceResponse.Builder()
        .say(new Say.Builder("Thanks for taking our survey.").build());

    return nextTwiMLQuestion(survey, voiceResponseBuilder).toXml();
  }

  String nextTwiMLQuestion(Survey survey) throws TwiMLException, UnsupportedEncodingException {
    return nextTwiMLQuestion(survey, null).toXml();
  }

  private VoiceResponse nextTwiMLQuestion(Survey survey, VoiceResponse.Builder twiml)
      throws TwiMLException, UnsupportedEncodingException {
    Question question = Server.config.getQuestions()[survey.getIndex()];

    return buildQuestionTwiML(survey, question, twiml);
  }

  private VoiceResponse buildQuestionTwiML(Survey survey, Question question,
      VoiceResponse.Builder twiml) throws TwiMLException, UnsupportedEncodingException {
    VoiceResponse.Builder response = twiml != null ? twiml : new VoiceResponse.Builder();
    Say say = new Say.Builder(question.getText()).build();
    response.say(say);
    // Depending on the question type, create different TwiML verbs.
    switch (question.getType()) {
      case "text":
        appendTextQuestion(survey, response);
        break;
      case "boolean":
        appendBooleanQuestion(response);
        break;
      case "number":
        appendNumberQuestion(response);
        break;
    }
    return response.build();
  }

  private VoiceResponse.Builder appendNumberQuestion(VoiceResponse.Builder twiml)
      throws TwiMLException {
    Say numInstructions =
        new Say.Builder("Enter the number on your keypad, followed by the #.").build();
    twiml.say(numInstructions);
    // Listen until a user presses "#"
    Gather numberGather = new Gather.Builder().finishOnKey("#").build();

    twiml.gather(numberGather);

    return twiml;
  }

  private VoiceResponse.Builder appendBooleanQuestion(VoiceResponse.Builder twiml)
      throws TwiMLException {
    Say boolInstructions =
        new Say.Builder("Press 0 to respond 'No,' and press any other number to respond 'Yes.'")
            .build();
    twiml.say(boolInstructions);

    // Listen only for one digit.
    Gather booleanGather = new Gather.Builder().numDigits(1).build();
    twiml.gather(booleanGather);

    return twiml;
  }

  private VoiceResponse.Builder appendTextQuestion(Survey survey, VoiceResponse.Builder twiml)
      throws TwiMLException, UnsupportedEncodingException {

    Say textInstructions = new Say.Builder(
        "Your response will be recorded after the tone. Once you have finished recording, press the #.")
            .build();
    twiml.say(textInstructions);
    Record text = new Record.Builder()
        .finishOnKey("#")
        .transcribe(true)
        .transcribeCallback(
            String.format("/interview/%s/transcribe/%s", urlEncode(survey.getPhone()), survey.getIndex())
        ).build();

    twiml.record(text);

    return twiml;
  }

  // Wrap the URLEncoder and URLDecoder for cleanliness.
  private String urlEncode(String s) throws UnsupportedEncodingException {
    return URLEncoder.encode(s, "utf-8");
  }

  String goodByeTwiMLMessage() throws TwiMLException {
    VoiceResponse voiceResponse = new VoiceResponse.Builder()
        .say(new Say.Builder("Your responses have been recorded. Thank you for your time!").build())
        .build();

    return voiceResponse.toXml();
  }
}

Let's see how to work with the transcription of a recorded response.

Using a Transcript of a Recorded Response

When Twilio finishes transcribing a free-form voice response, the application finds the corresponding survey and updates the response for that question to include the transcription. This makes it easier to skim the responses without listening to the full call.

package com.twilio.survey.controllers;

import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.util.HashMap;
import java.util.Map;

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import com.twilio.survey.Server;
import com.twilio.survey.models.Response;
import com.twilio.survey.models.Survey;
import com.twilio.survey.models.SurveyService;
import com.twilio.survey.util.IncomingCall;

import spark.Route;

public class SurveyController {
  private static SurveyService surveys = new SurveyService(Server.config.getMongoURI());

  // Main interview loop.
  public static Route interview = (request, response) -> {
    Map<String, String> parameters = parseBody(request.body());
    IncomingCall call = IncomingCall.createInstance(parameters);
    AbstractMessageFactory messageFactory = AbstractMessageFactory.createInstance(parameters);

    Survey existingSurvey = surveys.getSurvey(call.getFrom());
    if (existingSurvey == null) {
      Survey survey = surveys.createSurvey(call.getFrom());
      return messageFactory.firstTwiMLQuestion(survey);
    } else if (!existingSurvey.isDone()) {
      existingSurvey.appendResponse(new Response(call.getInput()));
      surveys.updateSurvey(existingSurvey);
      if (!existingSurvey.isDone()) {
        return messageFactory.nextTwiMLQuestion(existingSurvey);
      }
    }
    return messageFactory.goodByeTwiMLMessage();
  };


  // Results accessor route
  public static Route results = (request, response) -> {
    Gson gson = new Gson();
    JsonObject json = new JsonObject();
    // Add questions to the JSON response object
    json.add("survey", gson.toJsonTree(Server.config.getQuestions()));
    // Add user responses to the JSON response object
    json.add("results", gson.toJsonTree(surveys.findAllFinishedSurveys()));
    response.type("application/json");
    return json;
  };

  // Transcription route (called by Twilio's callback, once transcription is complete)
  public static Route transcribe = (request, response) -> {
    IncomingCall call = IncomingCall.createInstance(parseBody(request.body()));
    // Get the phone and question numbers from the URL parameters provided by the "Record" verb
    String surveyId = request.params(":phone");
    int questionId = Integer.parseInt(request.params(":question"));
    // Find the survey in the DB...
    Survey survey = surveys.getSurvey(surveyId);
    // ...and update it with our transcription text.
    survey.getResponses()[questionId].setAnswer(call.getTranscriptionText());
    surveys.updateSurvey(survey);
    response.status(200);
    return "OK";
  };

  // Helper methods

  // Spark has no built-in body parser, so let's roll our own.
  public static Map<String, String> parseBody(String body) throws UnsupportedEncodingException {
    String[] unparsedParams = body.split("&");
    Map<String, String> parsedParams = new HashMap<String, String>();
    for (int i = 0; i < unparsedParams.length; i++) {
      String[] param = unparsedParams[i].split("=");
      if (param.length == 2) {
        parsedParams.put(urlDecode(param[0]), urlDecode(param[1]));
      } else if (param.length == 1) {
        parsedParams.put(urlDecode(param[0]), "");
      }
    }
    return parsedParams;
  }

  public static String urlDecode(String s) throws UnsupportedEncodingException {
    return URLDecoder.decode(s, "utf-8");
  }
}

Let's take a look at how to store the caller's response.

Storing the Caller's Response

After each TwiML verb is executed and input is collected, Twilio sends that input to the application along with a request for more TwiML. The
/interview route in our controller handles this input and passes it to the model to update the survey. Using MongoDB's Morphia document-object mapper, we can update the question index, responses, and completeness status of the survey.

package com.twilio.survey.models;

import java.util.List;

import org.mongodb.morphia.Datastore;
import org.mongodb.morphia.Morphia;
import org.mongodb.morphia.query.UpdateOperations;

import com.mongodb.MongoClient;
import com.mongodb.MongoClientURI;
import com.twilio.survey.Server;

public class SurveyService {
  // MongoClient and Morphia instances must be accessible to the entire object, so a Datastore can
  // be built.
  MongoClient mongoClient;
  Morphia morphia;

  // An instance of Datastore must be accessible to the entire object, so all instance methods can
  // persist to and read from the datastore.
  Datastore datastore;

  // Constructor
  public SurveyService(MongoClientURI mongoURI) {
    try {
      // Create MongoDB drivers
      mongoClient = new MongoClient(mongoURI);
      morphia = new Morphia();

      // Ask the Morphia driver to scan the Models package for models.
      morphia.mapPackage("com.twilio.survey.models");

    } catch (Exception e) {
      // Catch any MongoDB configuration errors, and pass them back to STDERR.
      System.err.println(e.getMessage());
    } finally {
      // Create a datastore with the database name provided.
      datastore = morphia.createDatastore(mongoClient, Server.config.getMongoDBName());
    }
  }

  public SurveyService() {
    this(Server.config.getMongoURI());
  }

  // Find, Update, and Create -- database operations.
  public Survey getSurvey(String phone) {
    return datastore.find(Survey.class).field("phone").equal(phone).get();
  }

  public void updateSurvey(Survey survey) {
    UpdateOperations<Survey> updates = datastore.createUpdateOperations(Survey.class);
    updates.set("index", survey.getIndex());
    updates.set("responses", survey.getResponses());
    updates.set("done", survey.isDone());
    datastore.update(survey, updates);
  }

  public Survey createSurvey(String phone) {
    Survey existingSurvey = getSurvey(phone);
    if (existingSurvey == null) {
      Survey survey = new Survey(phone);
      datastore.save(survey);
      return survey;
    } else {
      return existingSurvey;
    }
  }

  public List<Survey> findAllFinishedSurveys() {
    return datastore.find(Survey.class).field("done").equal(true).asList();
  }
}

Finally, let's take a look at how to display our survey results.

Displaying Survey Results

In order to view our results, the completed surveys are serialized to JSON which then populates the dashboard. The dashboard is served at the root (/) URL of the application. We use jQuery to call this route via Ajax and display the survey results in charts on the home page.

package com.twilio.survey.controllers;

import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.util.HashMap;
import java.util.Map;

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import com.twilio.survey.Server;
import com.twilio.survey.models.Response;
import com.twilio.survey.models.Survey;
import com.twilio.survey.models.SurveyService;
import com.twilio.survey.util.IncomingCall;

import spark.Route;

public class SurveyController {
  private static SurveyService surveys = new SurveyService(Server.config.getMongoURI());

  // Main interview loop.
  public static Route interview = (request, response) -> {
    Map<String, String> parameters = parseBody(request.body());
    IncomingCall call = IncomingCall.createInstance(parameters);
    AbstractMessageFactory messageFactory = AbstractMessageFactory.createInstance(parameters);

    Survey existingSurvey = surveys.getSurvey(call.getFrom());
    if (existingSurvey == null) {
      Survey survey = surveys.createSurvey(call.getFrom());
      return messageFactory.firstTwiMLQuestion(survey);
    } else if (!existingSurvey.isDone()) {
      existingSurvey.appendResponse(new Response(call.getInput()));
      surveys.updateSurvey(existingSurvey);
      if (!existingSurvey.isDone()) {
        return messageFactory.nextTwiMLQuestion(existingSurvey);
      }
    }
    return messageFactory.goodByeTwiMLMessage();
  };


  // Results accessor route
  public static Route results = (request, response) -> {
    Gson gson = new Gson();
    JsonObject json = new JsonObject();
    // Add questions to the JSON response object
    json.add("survey", gson.toJsonTree(Server.config.getQuestions()));
    // Add user responses to the JSON response object
    json.add("results", gson.toJsonTree(surveys.findAllFinishedSurveys()));
    response.type("application/json");
    return json;
  };

  // Transcription route (called by Twilio's callback, once transcription is complete)
  public static Route transcribe = (request, response) -> {
    IncomingCall call = IncomingCall.createInstance(parseBody(request.body()));
    // Get the phone and question numbers from the URL parameters provided by the "Record" verb
    String surveyId = request.params(":phone");
    int questionId = Integer.parseInt(request.params(":question"));
    // Find the survey in the DB...
    Survey survey = surveys.getSurvey(surveyId);
    // ...and update it with our transcription text.
    survey.getResponses()[questionId].setAnswer(call.getTranscriptionText());
    surveys.updateSurvey(survey);
    response.status(200);
    return "OK";
  };

  // Helper methods

  // Spark has no built-in body parser, so let's roll our own.
  public static Map<String, String> parseBody(String body) throws UnsupportedEncodingException {
    String[] unparsedParams = body.split("&");
    Map<String, String> parsedParams = new HashMap<String, String>();
    for (int i = 0; i < unparsedParams.length; i++) {
      String[] param = unparsedParams[i].split("=");
      if (param.length == 2) {
        parsedParams.put(urlDecode(param[0]), urlDecode(param[1]));
      } else if (param.length == 1) {
        parsedParams.put(urlDecode(param[0]), "");
      }
    }
    return parsedParams;
  }

  public static String urlDecode(String s) throws UnsupportedEncodingException {
    return URLDecoder.decode(s, "utf-8");
  }
}

Pat yourself on the back! If you have configured one of your Twilio numbers to the application built in this tutorial, you should be able to take the survey and see the results under the root route of the application. We hope you found this sample application useful. 

Where to Next?

If you're hungry for more ways to work with Twilio APIs in Java, check out one of our other tutorials, like:

Appointment Reminders

Automate the process of reaching out to your customers in advance of an upcoming appointment.

Did this help?

Thanks for checking out this tutorial! If you have any feedback to share with us, we'd love to hear it. Connect with us on Twitter and let us know what you build!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.