Using Natural Language Processing for Better SMS Interfaces Using Twilio and Python’s TextBlob

Time to read: 6 minutes

June 25, 2014
Written by
Rob Spectre
Twilion

bill_and_teds_excellent_adventure_5

The International Telecommunications Union, the telecom agency for the United Nations, recently released some data suggesting for every 100 people on Planet Earth, 96 of them have a subscription to a cellular service.  That means there are nearly as many cell subscriptions out there as human beings – a stonking 6.8 billion.

Holy biscuits – that means there might well be more cell phones out there than people.

With Twilio, you can build apps with that touch every last one of them.  Voice and SMS are the only apps that come preinstalled on each of the devices those 6.8 billion people use.  However, with that many types of phones and languages out there to interface with, building interfaces that work for every human can be a fascinating engineering challenge.

Take SMS for example.  We’ve all come accustomed to text interfaces that ask for a KEYWORD in all caps and then take some action or deliver some information.  But with the wealth of tools at our fingertips as software people, surely we can do better than applications that hinge on single words or phrases in all caps like “Text HELP to do something useful.”

One of those tools is natural language processing – the field of computer science, artificial intelligence and linguistics dedicated to improving the way computers interpret human input.  NLP is an endlessly riveting and complex field that mixes very deep comprehension and dissection of the way humans speak in different languages with stonkingly labyrinthine statistical models to makes computers interact with us the way people do.  That rabbit hole goes deep and can definitely be intimidating for the mere web hacker, but fortunately the ecosystem around the field is producing some helpful open source projects that take the same discipline that powers Siri, Google Now, Cortana and Watson, and makes it accessible to the lay programmer.

One such project in the Python community is called TextBlob.  It’s a library I found through the excellent Pycoder’s Weekly aiming to provide a dead simple interface to many common natural language processing tasks.  In this post, we’ll use this tool to improve a few common SMS interfaces to create a more useful, more human experience for our Twilio SMS apps.

What You’ll Need

Here’s the ingredients list to cook up a tasty NLP souffle for our Twilio SMS app.

  • Twilio SMS – sign up for a free trial account here.
  • Python – the programming language our tutorial will be in.
  • Flask – a Python web microframework that will make it easy to accept incoming SMS messages.  Used here only for illustration, you’re welcome to use any web framework that works for you.
  • TextBlob – our natural language processing library we’ll use to improve our SMS interfaces.

Setting Up The Project

First we need to sign up for our free Twilio account.  Once your signup is complete and at your Twilio Account Dashboard, let’s click on the Numbers tab, select theTwilio phone number we would like to use for the project and configure it to point to the Python Twilio app we’ll be creating.

Next, we need to set up our project environment to get hacking.  First, we’ll install the Twilio Python module from the command line using Pip.

 

pip install twilio

 

Next we’ll install the two other Python packages  we’ll use for this project.  The first is Flask, a web microframework that makes building Twilio apps very quick.  The second is TextBlob, the library we’ll use for our natural language processing.

 

pip install flask
pip install textblob

 

Many NLP classifiers do depend on a collection of pre-classified data called a corpus.  TextBlob makes it easy to download the collections it relies on for natural language processing with a handy function:

 

python -m textblob.download_corpora

 

With our requirements installed, we’ll set up a quick Hello World app that we can modify to build our humanized SMS interfaces.  Here’s a quick, non-production example:

 

from flask import Flask
from flask import request

from twilio import twiml

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    response = twiml.Response()
    body = request.form['Body']

    response.message("You sent me: {0}".format(body))

    return str(response)

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

We’ll then run the Flask app to make it ready to receive an SMS message:

python app.py

Let’s send a text message to the Twilio phone number we configured up in the first step and see if it worked.

Excellent!  Now we’re ready to implement some natural language processing to improve the interfaces to our Twilio SMS apps.

Keeping Customers Happy with Analyzing Sentiment

A common application for NLP is sentiment analysis: determining programmatically whether user input is positive or negative.  Let’s create a quick stub app that simulates a service that provides customers support via texting.

 

from flask import Flask
from flask import request

from twilio import twiml

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    # Create the Twilio response
    response = twiml.Response()
    # Retrieve the body of the text message.
    body = request.form['Body']

    response.message("Thank you for texting Stallyns Support. What can "
                     "we help you with today?")

    return str(response)

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

Though our app doesn’t do anything yet, if we send a SMS to our project’s Twilio number, we can see a nice human response from our app.

To make the experience more personal, we could add some sentiment analysis to the application.  Based on whether or not the customer is happy or not, we can have our app respond accordingly.

Let’s add a little conditional logic to our app to supply different SMS replies based on the positivity or negativity of the user’s text message.

 

from flask import Flask
from flask import request

from twilio import twiml

# Add an import for the TextBlob library
from textblob import TextBlob

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    response = twiml.Response()
    body = request.form[‘Body’]

    # Use the body of the user's text message to create a new TextBlob object.
    text_blob = TextBlob(body)

    # Get sentiment of the user's statement.
    # >>> sentiment = text_blob.sentiment
    # >>> sentiment.polarity
    # 0.0
    sentiment = text_blob.sentiment

    # If the polarity of the sentiment is greater than zero, the statement is
    # positive.  Highest positivity is 1.0
    if sentiment.polarity > 0:
        response.message("That's great to hear.  Is there anything else we "
                         "can help you with today?")
    # If the polarity of the sentiment is less than zero, the statement is
    # negative.  Lowest negativity is -1.0.
    elif sentiment.polarity < 0:
        response.message("Oh no - we're sorry to hear that! How can we make "
                         "your experience better?")
    # If the polarity is 0.0, TextBlob was unable to determine the sentiment
    # of the statement.  In this case, we'll return a neutral response in turn.
    else:
        response.message("Thank you for that info.  What can I help you with?")

    return str(response)

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

 

Let’s give the app a try.  We’ll try emulating an unhappy customer.

Then try emulating a happy customer.

Then try a behavior that is a little more “meh.”


Party on – we can take wildly disparate input that we could not possibly anticipate and with just a few lines of code customize our responses for a more human experience.  These kind of determinations can have wide implications beyond just customer support.  Recording sentiment can improve SMS apps that collect survey data, handle logistics alerts like deliveries and even facilitate classroom instruction.

Forgiving Fat Fingers with Spelling Correction

One of the biggest issues with keyword SMS interfaces is the tendency for us humans to making spelling errors on the tiny keypads and touchscreens that power our texting.  While parsing for a keyword in all capital letters like HELP is really easy for programmers, it’s not so easy for users.

Many keypads and keyboards have tiresome caps lock mechanisms and even four characters can be easy to miss when you’re typing with a single thumb.  Let’s see if we can make a HELP menu for a Twilio SMS app more human by adding some NLP.

First, let’s build a HELP menu by looking for the HELP keyword and responding with a menu.

from flask import Flask
from flask import request

from twilio import twiml

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    # Construct the TwiML response.
    response = twiml.Response()
    # Collect the body of the text message from the user.
    body = request.form['Body']

    # If user sends HELP in the text message, display the HELP menu.
    if "HELP" in body.upper():
        response.message("Text EXCELLENT to party on. Text BOGUS to "
                         "to be most non-triumphant.")
    # If not, send a generic message.
    else:                
        response.message("Thank you for texting Stallyns. Text HELP for more "
                         "info.")

    return str(response)

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

Giving it a try with our phone, looks like we have the basic HELP keyword menu every Twilio SMS app should have.  But we also see the rub – my chubby thumb totally beefed typing HELP correctly and AutoCorrect only kicks in after a space or enter.  For one word keyword instructions, the smartphone often doesn’t correct before the message is already sent.  Totally bogus experience.

Let’s see if we can make this Twilio SMS app a little more forgiving of our human foibles with some NLP.

from flask import Flask
from flask import request

from twilio import twiml

from textblob import TextBlob

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    response = twiml.Response()
    body = request.form['Body']

    help_message = "Text EXCELLENT to party on.  Text BOGUS to be most " \
                   "non-triumphant."

    # If the user uses the keyword, give her the HELP menu.
    if "HELP" in body.upper():
        response.message(help_message)
    # If the explicit keyword is not found, check for spelling errors
    elif is_keyword_misspelled(body, "HELP"):
        response.message(help_message)
    # If keyword is still not found, return generic message.
    else:
        response.message("Thank you for texting Stallyns. Text HELP for more "
                         "info.")

    return str(response)

def is_keyword_misspelled(body, keyword):
    """Spell check each word in a message and see if it matched a keyword for
    our app.

    Args:
        body: A str representing the message you want to spellcheck.
        keyword: A str representing the keyword that might be misspelled.

    Returns:
        Bool: True if keyword is present in a spelling alternative in the
        message, False if not.

    """
    text_blob = TextBlob(body)

    # Get list of words in uppercase.
    for word in text_blob.words.lower():
        # Spellcheck each word.  Returns alternative spelling suggestions
        # and confidence score.
        suggestions = word.spellcheck()
        for suggestion in suggestions:
            # Check if confidence of the spelling alternative is over 80%.
            if suggestion[1] > 0.8:
                # Is it our keyword?
                if suggestion[0] == keyword.lower():
                    return True

    # If we don't find our keyword, return False
    return False

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

Now if we try our project app, we can see that we despite our frequent spelling errors, we still get back the keyword menu we’re intending to retrieve.

That’s a user experience that could cause even the most course button masher to break into air guitar spontaneously.

 

 

bill_and_teds_excellent_adventure_5
Let’s finish up showing how natural language processing can create better experiences for non-English texters as well.

Picking Up What Your Customer Is Putting Down With Language Detection

In SMS applications serving users of multiple languages, informational menus are often presented to users with language specific keywords.

For example, if we were serving customers through our Twilio SMS app in both English and Spanish, we might serve the HELP keyword for an English menu and the AYUDA keyword for a Spanish menu.

A stub for an app like this would like:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from flask import Flask
from twilio import twiml

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    response = twiml.Response()

    response.message(u"Text HELP for more information.  Texto de AYUDA para "
                     u"información en español.")

    return str(response)

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

 

But that’s not a great experience for the user, right?  In order to serve all the users, we have to make horrible UX decisions that bias user behavior, like what order we position the languages and what preference is given in the event of cognate collision.

As developers, we have all the information we need about the user’s language we need right in the text he/she is sending us.  Using NLP to detect the language, we can create the best personalization of all with our application by conforming our application dynamically to the language of the user.

Let’s take a stab at what a language detecting help menu would like like using TextBlob’s built-in methods for the Google Translate API.

 

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from flask import Flask
from flask import request

from twilio import twiml

from textblob import TextBlob

app = Flask(__name__)

@app.route('/sms', methods=['POST'])
def sms():
    response = twiml.Response()
    body = request.form['Body']

    # Create a TextBlob object using the user's input.
    blob = TextBlob(body)

    # Detect the language with TextBlob's Google Translate integration
    language = blob.detect_language()

    if language == 'es':
        response.message(u"Gracias por los mensajes de texto Stallyns usted. "
                         u"Texto de ayuda para más información.")
    else:
        response.message(u"Thank you for texting Stallyns. Text HELP for more "
                         u"info.")

    return str(response)

if __name__ == "__main__":
    # Since this is a development project, we will keep debug on to make our
    # lives easier. We would want to configure our app more conservatively
    # in production.
    app.debug = True
    app.run()

So if we try texting into our Twilio SMS app now with emulating both English-speaking and Spanish-speaking customers, we see a much more human personalization in each customer’s native tongue.

Most triumphant!

Playing More With NLP

Thanks for checking out how to use natural language processing to improve the SMS Interfaces of your Twilio apps.  We prototyped some examples of how NLP can be used to create more human experiences for your users.  We used sentiment analysis to customize our automated responses based on the user’s input.  We leveraged spelling correction to forgive the occasional mistakes that happen with SMS keywords.  And finally we used language to detection to ensure our app is literally speaking the same language as our customer.

Natural language processing is a way fun field of study and there are plenty more practical ways you can leverage it in your Twilio apps.  Here are some other projects I found helpful when exploring NLP:

I would love to see what you build with NLP and Twilio – you can find me on Twitter @dn0t or via email at rob@twilio.com.  Be excellent to each other!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.