Build an Urban Dictionary Emoji/Slang Sentiment Analyzer with Python and Twilio SMS

Time to read: 5 minutes

June 13, 2022
Written by
Johnny Nguyen
Twilion
Reviewed by
Niels Swimberghe
Twilion

build an urban dictionary emoji slang sentiment analyzer with python and twilio sms

Emojis and slang have, not only different nuances, but vastly different interpretations between groups of people. For instance, do your coworkers misappropriate the 🙂 emoji? How is it that 🙃 has more positive sentiment than 🙂?

In this tutorial, you will build a sentiment analyzer with Python and Twilio SMS using Urban Dictionary as a reference for the most recent slang.

Prerequisites

To continue with this tutorial, you will need:

Obtain a Twilio Phone Number

If you haven’t obtained a phone number from Twilio already, you can buy one for your account. Even if you are on a trial account, you are given trial credit on your balance to experiment with.

First, log into your account at https://console.twilio.com/. On the left-hand side of your dashboard, navigate to Phone Numbers > Manage > Buy a number.

Twilio dashboard with multi-level navigation on the left-hand side. The user clicks on the "Phone Numbers" > Manage > "Buy a number" link.

Find a suitable phone number you like on the Buy a Number page, and purchase it. 

Set up your project environment

Before you can dive into the code, you will need to set up your environment. First, you should create a parent directory that holds the project. Open the terminal on your computer, navigate to a suitable directory for your project, type in the following command, and hit enter.

mkdir urbandict-sms-project && cd urbandict-sms-project

As a part of good practices for Python, you should also create a virtual environment. If you are working on UNIX or macOS, run the following commands to create and activate a virtual environment.

python3 -m venv venv
source venv/bin/activate

However, if you are working on Windows, run these commands instead:

python -m venv venv
venv\bin\activate

After activating your virtual environment, you will install some Python packages. For this tutorial, you will use the following packages:

To install these packages, run this command:

pip install flask python-dotenv twilio urllib3 nltk

Afterwards, you need to install some necessary data from the Natural Language Toolkit. Type and enter python3 in the command line. This starts a python shell. Then, type and enter the following lines:

import nltk
nltk.download('vader_lexicon')
nltk.download('stopwords')
exit()

Your environment is set up. You are now ready to begin creating the application.

Create the Sentiment Analyzer

The Plan

It’s good practice to understand what the code should accomplish before coding it. Below are a sequence of actions that is expected for this program:

  • The user texts a search query to a phone number. The query can be an emoji, a word, or multiple words.
  • The application should search Urban Dictionary’s API for the query and obtain the definitions that resulted from the search.
  • The application analyzes the text of the definitions. It finds the polarity of the sentiment as well as the top 10 most frequent words.
  • The application texts this information back to the user.

The Code

Now you are ready to start coding the program. Open a new file in the urbandict-sms-project directory named app.py and paste the following code in the file:

from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse
from nltk.sentiment import SentimentIntensityAnalyzer
from nltk.corpus import stopwords
import json
from urllib.request import urlopen
from urllib.parse import quote

app = Flask(__name__)
sia = SentimentIntensityAnalyzer()

stop_list = stopwords.words("english")
stop_list += ["emoji", "you're", "it's", "used", ""]

def reply(text):
    resp = MessagingResponse()
    resp.message(text)
    return str(resp)

@app.route("/analyze", methods=['GET','POST'])
def analyze():
    body = request.form['Body'] # obtain text from user
    query = quote(body) # format special characters for URL query

    # search for the query in Urban Dictionary's API
    with urlopen(f'https://api.urbandictionary.com/v0/define?term={query}') as f:
        data = json.loads(f.read())

    # Concatenate each urban dictionary definition into one string and formatting
    # the text by removing special characters and lowercasing it
    definitions = ""
    for d in data['list']:
        definitions += ' ' + d['definition']
    definitions = definitions.translate({ord(x): ' ' for x in "[].?(),\"/;"})
    definitions = definitions.lower()

    # Counting and sorting words in the text (excluding stop words) with the
    # highest frequency
    keywords = {}
    for word in definitions.split(" "):
        if word not in stop_list:
            if word not in keywords:
                keywords[word] = 1
            else:
                keywords[word] += 1
    sorted_keywords = sorted(keywords, key=keywords.get, reverse=True)

    # Get and print polarity scores, print top 10 keywords
    scores = sia.polarity_scores(definitions)
    ret = f'Polarity Scores:\
        \nPositive {scores["pos"]}\
        \nNeutral {scores["neu"]}\
        \nNegative {scores["neg"]}\
        \nCompound {scores["compound"]}\n\n'
    ret += "Keywords:\n" + ", ".join(sorted_keywords[:10])

    return reply(ret)

There are a few things to note in the code. After importing the module and application, Flask and the SentimentIntensityAnalyzer are initialized. The SentimentIntensityAnalyzer uses VADER, a pretrained model from the natural language toolkit that performs sentiment analysis. In this case, the model analyzes the sentiment in the text from Urban Dictionary.

This application not only analyzes the polarity of what is queried, but also, it will keep track of frequent words that were mentioned. Since words that carry little meaning often occur most frequently (e.g. the word “the”), these words should be excluded from the count. A premade list of these words, known as stop words, are included in the nltk package. The code adds a few more stop words to the list and excludes them from the count.

The helper function reply(text) is used to create a TwiML MessagingResponse object so that the application can send a message back to the user.

In the analyze() function, the text message is obtained from the user. The text is then formatted and searched for in Urban Dictionary’s API, returning the definitions. The definitions are then concatenated together and analyzed for their polarity and word frequency. This information is then returned and messaged to the user.

Set up the ngrok tunnel and webhook

If you were to run the program prematurely, you would not receive a message from the phone number. There are two things that need to be addressed. First, you need to connect your local server to a publicly accessible URL by opening up a ngrok tunnel to your local machine. Second, you need to connect the Twilio phone number to the application by using a webhook.

After having installed ngrok on your machine, open another command prompt window, and run this command to open a tunnel to port 5000.

ngrok http 5000

Afterwards, your command prompt should something look like this:

Running ngrok in the command prompt, a list of session statuses is displayed, including a forwarding URL.

Next to the word Forwarding there should be a URL that points to https://localhost:5000. Your URL should be different than the one shown above. Copy that URL, as it will be used to set up the webhook. Leave this command prompt open.

On the left-hand side of your Twilio console, navigate to Phone Numbers > Manage > Active numbers. Clicking Active numbers will take you to the Active numbers page. Click the link for the phone number that you will be using for this application.

This will take you to the configure page for the phone number. Scroll down to Messaging and paste the URL you copied from before to the A MESSAGE COMES IN section, and add “/analyze” at the end of the URL. So your URL should be something like https://your-ngrok-url.ngrok.io/analyze and should be pasted like the example below:

Twilio Phone Number configuration form where the phone number is configured to send text messages to a webhook with the ngrok Forwarding URL.

Click Save at the bottom of the page to update the number’s settings. You are now finally ready to run the application.

Run the Application

To run the application, run this command in the terminal.

flask run


If you are using macOS, there may be an SSL certificate verification error if you have not installed the certificates already. If so, click into the Python version you are using in the Applications folder, and click install Certificates.command. Run your application again.

Afterwards, your application should be working properly. Text a search query to the Twilio phone number, and a reply should appear.

SMS conversation where "hello world" was sent and polarity scores and keywords related to "hello world" was responded with.

SMS conversation with one participant sending an emojis and the other responding with polarity scores and keywords calculated from the emoji.

Conclusion

Congratulations on building an application with Twilio SMS. Your application not only receives a text message, but also, it analyzes the sentiment of the text and replies with the results of its analysis. Feel free to take any liberties you want with this application. Possibilities include (but are not limited to) changing the URL to a different website instead of Urban Dictionary, adding more functionality to the application, and sharing this application with others.

Excited to build more with Twilio and Python? Check out the resources below and learn more:

Johnny Nguyen is an intern developer on Twilio’s Developer Voices Team. He loves creating ironic coding projects for others to learn and enjoy. When he’s not napping or watching TikTok, he can be reached at ngnguyen [at] twilio.com.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.