Automated Survey with Python and Flask

Time to read: 4 minutes

January 10, 2017
Written by
Andrew T. Baker
Twilion
Reviewed by
Paul Kamp
Twilion
Jose Oliveros
Contributor
Opinions expressed by Twilio contributors are their own
Orlando Hidalgo
Contributor
Opinions expressed by Twilio contributors are their own
Kat King
Twilion
Brianna DelValle
Twilion

automated-survey-python-flask

Have you ever wondered how to create an automated survey that can be answered over phone or SMS?

This tutorial will show how to do it using Twilio's API.

Here's how it works at a high level

  1. The end user calls or sends an SMS to the survey phone number.
  2. Twilio gets the call or SMS and makes an HTTP request to your application asking for instructions on how to respond.
  3. Your web application instructs Twilio (using TwiML) to Gather or Record if it receives voice input. If using SMS it will prompt for text input with Message.
  4. After each question is answered, Twilio makes another request to your server with the user's input. That input is stored in the application's database.
  5. After storing the answer, our server will instruct Twilio to Redirect the user to the next question or finish the survey.

Creating a Survey

For your convenience, the application's repository already includes one survey that can be loaded into the database.

You can modify the survey questions by editing the survey.json file located in the root of the repository and re-running the app's dbseed command:

$ python manage.py dbseed

Editor: this is a migrated tutorial. Clone the original code from https://github.com/TwilioDevEd/automated-survey-flask/

from .models import Survey, Question
import json


def survey_from_json(survey_json_string):
    survey_dict = json.loads(survey_json_string)
    survey = Survey(title=survey_dict['title'])
    survey.questions = questions_from_json(survey_json_string)
    return survey


def questions_from_json(survey_json_string):
    questions = []
    questions_dicts = json.loads(survey_json_string).get('questions')
    for question_dict in questions_dicts:
        body = question_dict['body']
        kind = question_dict['type']
        questions.append(Question(content=body, kind=kind))
    return questions

Let's take a moment to understand the flow of a Twilio-powered survey as an interview loop.

The Interview Loop

The user can answer a question for your survey over the phone by either their phone's keypad or by voice. After each interaction Twilio will make an HTTP request to your web application with either the string of keys the user pressed or a URL to a recording of their voice input.

For SMS surveys the user will answer questions by replying with another SMS to the Twilio number that sent the question.

It's up to the application to process, store and respond to the user's input.

Let's dive into this flow to see how it actually works.

Configuring a Twilio Number

To initiate the interview process, we need to configure one of our Twilio numbers to send our web application an HTTP request when we get an incoming call or SMS.

Click on one of your numbers and configure Voice and Message URLs that point to your server. In our code, the route is /voice for Voice and /message for Messaging.

Automated Survey Webhook Setup

If you don't already have a server configured to use as your webhook, ngrok is a great tool for testing webhooks locally.

Next, we will see how to handle requests to our webhooks.

Responding to a Twilio Request

Right after receiving a call or SMS, Twilio will send a request to the URL specified in that phone number's configuration.

The endpoint used for a call is the /voice endpoint. It will check to see if we have a survey with questions to be answered, welcome the user, and redirect them to the first question using Twilio's <Redirect> verb.

The /message endpoint will receive each SMS, welcome the user, and redirect them to the proper question. If the SMS request has a question_id variable on its session then we redirect to our answer endpoint to store the answer. We'll dive into this routing more thoroughly later on. For now, imagine that we are going to the first survey question after welcoming the user.

from . import app
from .models import Survey
from flask import url_for, session
from twilio.twiml.voice_response import VoiceResponse
from twilio.twiml.messaging_response import MessagingResponse


@app.route('/voice')
def voice_survey():
    response = VoiceResponse()

    survey = Survey.query.first()
    if survey_error(survey, response.say):
        return str(response)

    welcome_user(survey, response.say)
    redirect_to_first_question(response, survey)
    return str(response)


@app.route('/message')
def sms_survey():
    response = MessagingResponse()

    survey = Survey.query.first()
    if survey_error(survey, response.message):
        return str(response)

    if 'question_id' in session:
        response.redirect(url_for('answer', question_id=session['question_id']))
    else:
        welcome_user(survey, response.message)
        redirect_to_first_question(response, survey)
    return str(response)


def redirect_to_first_question(response, survey):
    first_question = survey.questions.order_by('id').first()
    first_question_url = url_for('question', question_id=first_question.id)
    response.redirect(url=first_question_url, method='GET')


def welcome_user(survey, send_function):
    welcome_text = 'Welcome to the %s' % survey.title
    send_function(welcome_text)


def survey_error(survey, send_function):
    if not survey:
        send_function('Sorry, but there are no surveys to be answered.')
        return True
    elif not survey.has_questions:
        send_function('Sorry, there are no questions for this survey.')
        return True
    return False

Let's see how we can use a Controller to handle and maintain the state of the survey.

Question Controller

This endpoint checks if the incoming request is from an SMS or a Call and builds a survey question as a TwiML response. Each type of question and interaction (Call or SMS) will produce different instructions on how to proceed. For instance, we can record a voice message or gather a key press during a call, but we can't do the same for text messages.

When the user interacts with our survey over SMS we don't have something like an ongoing call session with a well defined state. Since all SMS requests will be sent to the /message main endpoint, it becomes harder to know if an SMS is answering question 2 or 20. To solve that, we can use Twilio Cookies to keep track of what question is being answered at the moment. This is done by setting a question_id session key, leaving Flask to handle cookie management.

from . import app
from twilio.twiml.voice_response import VoiceResponse
from twilio.twiml.messaging_response import MessagingResponse
from .models import Question
from flask import url_for, request, session


@app.route('/question/<question_id>')
def question(question_id):
    question = Question.query.get(question_id)
    session['question_id'] = question.id
    if not is_sms_request():
        return voice_twiml(question)
    else:
        return sms_twiml(question)


def is_sms_request():
    return 'MessageSid' in request.values.keys()


def voice_twiml(question):
    response = VoiceResponse()
    response.say(question.content)
    response.say(VOICE_INSTRUCTIONS[question.kind])

    action_url = url_for('answer', question_id=question.id)
    transcription_url = url_for('answer_transcription', question_id=question.id)
    if question.kind == Question.TEXT:
        response.record(action=action_url, transcribe_callback=transcription_url)
    else:
        response.gather(action=action_url)
    return str(response)


VOICE_INSTRUCTIONS = {
    Question.TEXT: 'Please record your answer after the beep and then hit the pound sign',
    Question.BOOLEAN: 'Please press the one key for yes and the zero key for no and then'
    ' hit the pound sign',
    Question.NUMERIC: 'Please press a number between 1 and 10 and then'
    ' hit the pound sign',
}


def sms_twiml(question):
    response = MessagingResponse()
    response.message(question.content)
    response.message(SMS_INSTRUCTIONS[question.kind])
    return str(response)


SMS_INSTRUCTIONS = {
    Question.TEXT: 'Please type your answer',
    Question.BOOLEAN: 'Please type 1 for yes and 0 for no',
    Question.NUMERIC: 'Please type a number between 1 and 10',
}

Let's see how the response is built.

Building Our TwiML Verbs

If the survey question is "numeric" or "boolean" (yes/no) in nature, then we use the <Gather> verb. However, if we expect the user to record a free-form answer we use the <Record> verb. Both verbs take an action attribute.

Twilio will use this attribute to define our answer endpoint to use as a callback. That endpoint will be responsible for receiving and storing the caller's answer.

During the Record verb creation, we also ask for a Transcription. Twilio will process the recording and extract useful text, making a request to our transcription endpoint when the transcription is complete.

from . import app
from twilio.twiml.voice_response import VoiceResponse
from twilio.twiml.messaging_response import MessagingResponse
from .models import Question
from flask import url_for, request, session


@app.route('/question/<question_id>')
def question(question_id):
    question = Question.query.get(question_id)
    session['question_id'] = question.id
    if not is_sms_request():
        return voice_twiml(question)
    else:
        return sms_twiml(question)


def is_sms_request():
    return 'MessageSid' in request.values.keys()


def voice_twiml(question):
    response = VoiceResponse()
    response.say(question.content)
    response.say(VOICE_INSTRUCTIONS[question.kind])

    action_url = url_for('answer', question_id=question.id)
    transcription_url = url_for('answer_transcription', question_id=question.id)
    if question.kind == Question.TEXT:
        response.record(action=action_url, transcribe_callback=transcription_url)
    else:
        response.gather(action=action_url)
    return str(response)


VOICE_INSTRUCTIONS = {
    Question.TEXT: 'Please record your answer after the beep and then hit the pound sign',
    Question.BOOLEAN: 'Please press the one key for yes and the zero key for no and then'
    ' hit the pound sign',
    Question.NUMERIC: 'Please press a number between 1 and 10 and then'
    ' hit the pound sign',
}


def sms_twiml(question):
    response = MessagingResponse()
    response.message(question.content)
    response.message(SMS_INSTRUCTIONS[question.kind])
    return str(response)


SMS_INSTRUCTIONS = {
    Question.TEXT: 'Please type your answer',
    Question.BOOLEAN: 'Please type 1 for yes and 0 for no',
    Question.NUMERIC: 'Please type a number between 1 and 10',
}

Now let's see what to do once we receive a response.

Handling Responses

After the user has finished submitting their answers, Twilio sends us a request explaining what happened and asking for further instructions.

At this point, we need to recover data from Twilio's request parameters (extract_content handles this) and store it in the database.

Recovered parameters vary according to what we asked in our questions:

  • Body contains the text message from an answer sent via SMS.
  • Digits contains the keys pressed for a numeric question.
  • RecodingUrl contains the URL for a recorded message.
  • TranscriptionText contains the text of a voice recording transcription.

Finally we redirect to our Question controller, which will ask the next question in the loop. This is done in the redirect_twiml function.

from . import app, db
from .models import Question, Answer
from flask import url_for, request, session
from twilio.twiml.voice_response import VoiceResponse
from twilio.twiml.messaging_response import MessagingResponse


@app.route('/answer/<question_id>', methods=['POST'])
def answer(question_id):
    question = Question.query.get(question_id)

    db.save(
        Answer(
            content=extract_content(question), question=question, session_id=session_id()
        )
    )

    next_question = question.next()
    if next_question:
        return redirect_twiml(next_question)
    else:
        return goodbye_twiml()


def extract_content(question):
    if is_sms_request():
        return request.values['Body']
    elif question.kind == Question.TEXT:
        return 'Transcription in progress.'
    else:
        return request.values['Digits']


def redirect_twiml(question):
    response = MessagingResponse()
    response.redirect(url=url_for('question', question_id=question.id), method='GET')
    return str(response)


def goodbye_twiml():
    if is_sms_request():
        response = MessagingResponse()
        response.message("Thank you for answering our survey. Good bye!")
    else:
        response = VoiceResponse()
        response.say("Thank you for answering our survey. Good bye!")
        response.hangup()
    if 'question_id' in session:
        del session['question_id']
    return str(response)


def is_sms_request():
    return 'MessageSid' in request.values.keys()


@app.route('/answer/transcription/<question_id>', methods=['POST'])
def answer_transcription(question_id):
    session_id = request.values['CallSid']
    content = request.values['TranscriptionText']
    Answer.update_content(session_id, question_id, content)
    return ''


def session_id():
    return request.values.get('CallSid') or request.values['MessageSid']

Finally, let's see how to visualize the results.

Displaying the Survey Results

For this route we simply query the database for our survey answers using SQLAlchemy and then display the information within a template. We show a panel for every question from the survey, and inside each panel we list the responses from the different calls.

You can access this page in the applications root route.

from . import app
from . import question_view  # noqa F401
from . import answer_view   # noqa F401
from . import survey_view   # noqa F401
from flask import render_template
from .models import Question


@app.route('/')
def root():
    questions = Question.query.all()
    return render_template('index.html', questions=questions)

That's it!

If you have configured one of your Twilio numbers to the application built in this tutorial, you should be able to take the survey and see the results under the root route of the application. We hope you found this sample application useful. 

Where to Next?

If you're a Python developer working with Twilio, you might enjoy these other tutorials:

Gather User Input via Keypad (DTMF Tones)

Gather user input during a phone call through the phone's keypad (using DTMF tones).

Receive and Reply to SMS and MMS Messages

Use Programmable SMS to respond to incoming SMS messages in your web application.

Did this help?

Thanks for checking this tutorial out! If you have any feedback to share with us, we'd love to hear it. Connect with us on Twitter and let us know what you build!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.