Calculating Net Promoter Score with Python and SMS

On the Twilio Developer Evangelism team we’re constantly seeking quantitative feedback to improve how we work. In 2014, I began collecting feedback via SMS to calculate Net Promoter Score on my conference talks as one quantitative measure for how talks were received. For example, at the end of my DjangoCon 2014 talk you can see in the video how I asked the audience to text in a score based on how likely they were to recommend that talk to a friend or colleague.

In this blog post, we’ll walk through how you can also obtain quantitative data on your events by requesting scores from the audience via SMS and calculating Net Promoter Score with a Python script.

What is Net Promoter Score?

Before we get started with the code, let’s clarify what Net Promoter Score is and why it’s useful as a measurement tool. Net Promoter Score (NPS) is a quantitative metric used to gauge loyalty. NPS is often used by companies to determine customers’ loyalty to the firm. In our case, we want to use NPS to quantitatively measure an audience’s reaction to a talk or event. I’ve personally found NPS useful for my talks in determining if a talk is close to my expectations or needs to be entirely reworked.

There is a specific phrase required to accurately collect NPS. In our case for scoring a technical talk, this phrase is “How likely is it that you would recommend this talk to a friend or colleague?”. Scores are then collected from the responders on a 0-10 scale. A response of 0 is the low end and means “not likely at all” while a response of 10 is the high end, which means “very likely”.

The responders’ scores are categorized into promoters, detractors and passives. We can calculate NPS by subtracting the percentage of detractors, as measured by those who respond with a score of 0 to 6, from the percentage of promoters who score the talk either a 9 or 10.

Let’s work through an example where we have 30 responders. Five people score the talk from 0-6, 15 score the talk a 7 or an 8, and 10 score the talk either a 9 or a 10. With these scores we have 5 detractors, 15 passives and 10 promoters. The final NPS result is determined by calculating

(10 promoters / 30 responses) – (5 detractors / 30 responses)

.333 – .166 = .167

The .167 is then multiplied by 100 to convert it into the -100 to +100 range for a +16.7 NPS result.

Now that we know how to calculate the score, how do we interpret the results? A negative NPS score means the talk needs work, a positive NPS score is considered good and anything around or above the +50 NPS range is excellent. Scores above +50 NPS are rare, especially with software developers who tend to be more critical than other audiences. If you’re in positive territory you’re doing good work.

Now that we understand NPS and and how it’s calculated, let’s get into the details of how to collect NPS scores from incoming text messages.

What We’ll Need

We’ll need four items for calculating NPS via SMS:

Ensure you have an upgraded Twilio account to be ready to begin the next section of this post.

Setting Up Our Phone Number and Flask App

Head to the Twilio numbers page and click the “Buy a number” button if you don’t already have a phone number available.

When you click on the “Buy a number” button you’ll see the numbers search screen. I recommend buying a number with the area code of the location you’ll be giving a talk or running an event in. In my case, I’m located in Washington, D.C. so I searched for the D.C. area code 202.

Buy one of the numbers and when it’s confirmed you’ll see the following success modal.

Before we can configure the number on the next screen though, we need to set up an endpoint with a Flask application to tell Twilio how to respond to incoming messages.

There’s a few ways to set up and deploy the Flask application to handle incoming SMS responses from Twilio. The first way is to deploy our GitHub repository to Heroku. You can even push this handy deploy button below to get it set up for you:

Deploy this app to Heroku

Another way to handle the Flask application is to write the code or clone the repository then deploy it locally. We can then use a localhost tunnel to respond to inbound Twilio SMS messages. If you want to work through this approach and understand the code, read on. If you used the Heroku deploy button step down to the numbers screen below.

A best practice when working with Python is to isolate each project’s application dependencies from the other projects using virtualenv. Virtualenv installation depends on your operating system, so here are guides for Windows, Mac OS X and Ubuntu Linux users.

Create a new virtualenv to separate your Python dependencies from other apps you’re working on.

virtualenv nps

Be sure to activate the virtualenv before we install the dependencies.

source nps/bin/activate

The dependencies we need for this project are the Twilio helper library and Flask. To install this dependency run the following pip command on the command line. If you need to install pip, here are a few more specific installation guides for Windows, Mac OS X and Linux.

pip install twilio flask

Now your environment is ready to execute Python code that relies on Flask and the Twilio helper library, so let’s write the code to make it work.

Create a new file named app.py for our Flask application and add the following contents.

The above code creates a Flask web application and allows us to run it from the command line. Within the twilio_response function, we use the Twilio Python helper library to construct a response message that responds to an HTTP POST request with the message of your choice. I often include my contact information in there along with a link to the presentation if available.

Save the file and we can run the Flask application using the following command.

python app.py

The Flask server should display the following message if it starts up successfully through the above command.

* Running on http://127.0.0.1:5000/

To access the Flask application that’s running on your computer from other computers you’ll need to create a localhost tunnel using a tool such as Ngrok.

The reason why a localhost tunnel is necessary is that your development machine is likely behind a router that won’t allow direct inbound connections to your Flask application. We can get around this localhost issue with a localhost tunnel service, one of which is Ngrok. Sign up for Ngrok, download the Ngrok tunnel application and install it on your development machine.

Fire up Ngrok on port 5000 where our Flask application is running with the following command. See this configuring Ngrok post if you’re running Windows. On Linux and Mac OS X Ngrok can be run with the following command when you’re in the directory that Ngrok is located in.

./ngrok 5000

Ngrok will load up a screen like the one in the following screenshot. Take note of the unique https:// forwarding URL as we’ll need that again in a minute to configure our Twilio number’s incoming message handling URL.

The Flask application running on port 5000 can now be accessed through the forwarding URL set up by Ngrok.

We’re now ready to set up the Twilio number. Head back to the Twilio web page and press the “Setup number” button for the number you purchased. A configuration page with default values for voice and messaging will come up and look like the following screen.

The messaging text box, highlighted with a dotted green rectangle in the picture above, tells Twilio where to send an HTTP POST with the details of any inbound SMS sent to this number. If you used the Deploy to Heroku button, copy the URL for the application plus the /message endpoint. For example, if Heroku generated the app name of serene-beyond-5838 for your deployment, you would fill in the following URL into the Messaging Request URL text box.

https://serene-beyond-5838.herokuapp.com/message

If instead you used Ngrok, copy the Ngrok forwarding URL and paste it into that text box. Press the Save button and your setup should look similar to the following screen.

With our Flask application serving the response through Ngrok or Heroku and the messages webhook for our number set up, now anyone who sends a text message to the number will see the following response.

We’re set up to respond to inbound feedback text messages so let’s see how this would work with an audience.

Gathering Feedback Scores

Now we can gather responses from the audience, but we need to set the context for people so they know the feedback scale. I recommend putting up a slide in your presentation with the phone number that looks something like the following.

At the end of the talk or event, put up a slide like the one above and ask the audience to text their score to the number on the screen. Make sure to read the text and phone number while asking the audience for their honest score. Let folks know in your request that you’re looking for quantitative metrics to help improve the talk for future audiences.

Once the audience’s responses are captured and the talk is over we can shift our focus to analyzing the input.

Calculating Net Promoter Score

With our SMS responses and a bit of Python we can calculate NPS through the Twilio API and a Python script. Create a new file named nps.py to hold the Net Promoter Score code. The retrieval, parsing, filtering and displaying of the NPS results will be handled with four functions in this Python file.

At the top of the script we have our imports. These libraries are used for parsing arguments, extracting scores with regular expressions and handling dates and times. The last import is for the Twilio Python helper library.

Next up are two variables which set up our Twilio credentials. We can set the account SID and authentication token here by replacing them in this Python file or set the environment variables TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN with values found at the top of your Twilio account dashboard.

If you set the environment variables then the above variables can be left alone. Otherwise grab the account SID and auth token from the Twilio dashboard and replace the strings in your code with your credentials.

Next up in the script is the get_messages function which handles retrieval of messages from Twilio via an HTTP call performed by the Twilio Python helper library.

In get_messages we iterate through the messages sent to the phone number on a specific date. We store all messages associated with a phone number in an array. We’ll use the array to try and parse out a score from at least one of those messages. In most cases there will only be a single message from a phone number but occasionally people will text in several times with a score. When we filter the scores later we’ll only keep a single score per number so the results are not skewed by a single person texting in many scores from the same number.

The filter_scores function shown below loops through each message in the array and parses a score out from a message. As soon as we find a score we break from the array so that we do not count more than one score per phone number.

Inbound messages are not always clean numbers between 0 and 10, therefore we use a regular expression to extract out a whole number or one with a decimal place from the beginning of the message.

Next we need a to calculate the NPS and output the scores. In this script there are two functions, one named output_scores and another named calculate_nps. The main function calls output_scores to print a few strings to the command line with the number of responses received, calculated NPS and the breakdown of responses per score.

The calculate_nps function handles the NPS calculation by counting the promoter responses (9s and 10s) and the detractors (0s through 6s). Then the proportion of detractors is subtracted from the proportion of promoters. The return value is multiplied by 100 because NPS is in the range from 0 to 100.

Finally, let’s take a look at the main function that is invoked when the script is run from the command line.

In the above code we parse command line arguments for the phone number to search for the response SMS messages. A second optional argument specifies the date of the event and uses today’s date if no argument is passed in. We then instantiate the Twilio Python helper library with account credentials in the script or from environment variables.

Once we have the command line arguments and an instantiated helper library we invoke the get_messages function which issues an HTTP call to Twilio to pull down the messages for a specific date. The Python dictionary of messages returned is then filtered for unique numbers and numeric responses by the filter_scores function. If there is at least one score extracted from filtering the scores we display the NPS scores by calling the output_scores function.

That is all the code from the script. Here’s the entire script again, this time put together in a single code block.

Let’s give it a try now. Run it with the following command but replace the phone number below with your own response phone number.

$ python nps.py +12027336150

When we run the above command with our response number we’ll see output like the following assuming we had responses from the current date.

If instead we want to look at a past event we can use the --date argument with a date in the YYYY-MM-DD format. For example,

python nps.py +12027336150 --date 2015-01-15

With this output we have the final Net Promoter Score and a simple count for each score from 0-10. The breakdown by score helps to give a sanity check that the output is correct and that you have a good number of responses from the audience. For example, in talks I give with about 100 people I generally obtain 70-80 responses from unique phone numbers.

Quantitative Feedback on Technical Talks

If you’re looking to obtain quantitative feedback on your technical talks NPS is a good way to gain that insight. As a software developer I find that after a talk it greatly helps to have some numeric way to assess whether the event was worth the audience’s time or not. Then I can improve the talk based on qualitative feedback I gain from talking to people from the audience afterwards.

It would be great to hear how you’re using this SMS feedback technique in your own talks or if you use a different metric. Contact me and let me know via one of the following channels:

  • What a thoughtful post. Thanks Matt, We are totally going to use this to create metrics for interactions. This type of analysis adds tons of value to helping to identify real relationships.

    • Thank you! I began using this SMS feedback collection approach at my technical talks about six months ago. It’s really helped me put some numbers behind whether a talk needs to be completely revised or just iterated on. Let me know how it goes with your events & interactions.

      • Will post our blog post here. As soon as we have data!

  • Thanks. Great to see this in Javascript/Express