Image recognition in Python with the Clarifai API and Twilio MMS

Image recognition can seem like a pretty daunting technical challenge. Scraping images to use as training data for a machine learning model stresses me out. That’s where Clarifai comes in. This API is great for implementing image recognition so you can focus on the core functionality of what you are building.

Let’s build a Flask application in Python with Twilio MMS to receive picture messages over a phone number and respond with relevant keywords from Clarifai’s image recognition API.

Setting up your environment

Before moving on, make sure to have your Python environment setup. Getting everything working correctly, especially with respect to virtual environments, is important for isolating your dependencies if you have multiple projects running on the same machine.

You can also run through this guide to make sure you’re good to go before moving on.

Installing dependencies

Now that your environment is set up, you’re going to need to install the libraries we’ll use for this app:

First, navigate to the directory where you want this code to live and run the following command in your terminal with your virtual environment activated to install these dependencies:

Image recognition with Clarifai

Let’s start by writing a module to interact with the Clarifai API. Before being able to use the Clarifai API, you’ll have to make an account. Once you have an account, you’ll need to create an application so you have an API key to use. You can name your application whatever you want.

ClarifaiAPIKey.gif

Once you have an application, you need to set the following environment variable so the Clarifai Python module can use your API key to authenticate:

Now you can start writing some code. Create a file called tags.py and enter the following code:

What we’re doing here is defining a function that hits the Clarifai API with an image URL and returns all of the tags or “concepts” associated with that image. Try running this code with an image of your choice from the Internet and seeing what happens. I’m going to use this picture from a show my old band played in Philly back in 2012.

394057_328648487179283_294009412_n.jpg

Append the line print('n'.join(get_relevant_tags('*image_url*'))) to your code and run it:

You should see some results printing to your terminal. Here’s some of what I got. It’s interesting that Clarifai was basically able to figure out that this picture was of a group of people taken at a concert:

Screen Shot 2017-06-28 at 4.49.09 PM.png

You can delete that line you just added before moving on.

Setting up your Twilio account

Before being able to respond to picture messages, you’ll need a Twilio phone number. You can buy a phone number here.

Your Flask app will need to be visible from the internet in order for Twilio to send requests to it. We will use ngrok for this, which you’ll need to install if you don’t have it. In your terminal run the following command:

This provides us with a publicly accessible URL to the Flask app. Configure your phone number as seen in this image:

Screen Shot 2016-07-14 at 10.54.53 AM.png
You are now ready to send a text message to your new Twilio number.

Building the Flask app

Now that you have a Twilio number and are able to grab relevant tags and keywords associated with an image, you want to allow users to text a phone number with their own pictures for your code to analyze.

Let’s create our Flask app. Open a new file called app.py and add the following code:

We only need one route on this app: /sms to handle incoming text messages.

Run your code with the following terminal command:

Now take a selfie (or any picture) and send it to your Twilio phone number to see if it will recognize what’s inside!

How does this all work?

With this app running on port 5000, sitting behind our public ngrok URL, Twilio can see your application. Upon receiving a text message:

  1. Twilio will send a POST request to /sms.
  2. The sms_reply function will be called.
  3. The URL to the image in the text message is passed to our tags module
  4. A request to the Clarifai API will be made, receiving a response with keywords associated with our image.
  5. Your /sms route responds to Twilio’s request telling Twilio to send a message back with the tags we received from the Clarifai API.

For more Clarifai related fun, check out this post on how to hack your gift giving. If you’re interested in trying your hand at image recognition on your own with OpenCV, you can check out this blog post written by Megan Speir.

Feel free to reach out if you have any questions or comments or just want to show off the cool stuff you’ve built.