Build the future of communications.
Start building for free

Website SMS Alerts with the Plushcap Python Package: Part 1

plushcap-banner

When your website crashes you can either hear about it from your alerting software or an angry customer. That’s why a variety of hosted services exist to monitor websites.

However, for small websites you often want a simple open source project that can be set up in a couple of minutes to get the monitoring job done. In this post we’ll walk through the creation of a monitoring and alerting Python package.

This project is called Plushcap. What’s a Plushcap? It’s a species of tiny bird found in South American countries such as Argentina and Peru.

We’re going with the name Plushcap for a few reasons. First, the name is available on PyPI so there will not be any naming conflicts. Second, there are no projects named “plushcap” on GitHub in any language so we won’t be stepping on anyone else’s open source project name. Third and most importantly, I’m a huge fan of the amazing Pelican static website generator. Plushcap is a nod to that other P-bird-named open source project.

Many blog posts just walk through coding an app. This post will go a step farther and walk through the entire lifecycle of a Python open source project from initial code structure to making Plushcap available on PyPI.

Setting up our Initial Python Package

There’s some boilerplate code required for a Python project to be installable via PyPI. We can use the wonderful cookiecutter tool to get our initial codebase together. Cookiecutter has some external dependencies such as Jinja2 so I create a separate virtualenv just for these initial bootstrapping steps.

$ virtualenv --no-site-packages ~/Envs/cookie

Next we activate the virtualenv.

$ source ~/Envs/cookie/bin/activate

Install cookiecutter.

$ pip install cookiecutter

Now let’s create a directory where our project will be stored along with the requisite files for a Python package. Our input is bolded.

(cookie)~/devel/py$ cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git
Cloning into 'cookiecutter-pypackage'...
remote: Reusing existing pack: 377, done.
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 383 (delta 0), reused 1 (delta 0)
Receiving objects: 100% (383/383), 59.93 KiB, done.
Resolving deltas: 100% (187/187), done.
full_name (default is "Audrey Roy")? Matt Makai
email (default is "audreyr@gmail.com")? mmakai@twilio.com
github_username (default is "audreyr")? makaimc
project_name (default is "Python Boilerplate")? Plushcap
repo_name (default is "boilerplate")? plushcap
project_short_description (default is "Python Boilerplate contains all the boilerplate you need to create a Python package.")? Plushcap monitors websites and alerts people via text or phone call if there is a problem.
release_date (default is "2014-01-11")? 2014-07-06
year (default is "2014")?
version (default is "0.1.0")?
(cookie)~/devel/py$

What did cookiecutter just do? It cloned the cookiecutter-pypackage.git repository and inserted the input values into files in our Python package directory named plushcap. We now have a directory with the following files.

  • AUTHORS.rstreStructuredText file with a list of package author’s names and optionally their email addresses 
  • CONTRIBUTING.rst – guidelines for open source contributions to this project 
  • HISTORY.rst – placeholder file for documenting changes for each version of the library 
  • Makefile – convenient shortcut commands with the make program for packaging, testing and building the package and documentation 
  • LICENSE – boilerplate legalese with the BSD license 
  • MANIFEST.in – enumerates the files to include in the Python package distributable when it is built 
  • README.rst – the first documentation file for others to review when they encounter the project 
  • requirements.txt – application dependencies for developers working on the Plushcap library 
  • setup.cfgsetup configuration file 
  • setup.pyPython setup script for packaging 
  • tox.ini – configuration for the tox automation project 
  • .travis.yml – configuration file for Travis CI automated builds that are free for open source projects 
  • .gitignore – Python-specific Git ignore file for files we do not want to include in the package’s Git repository 

We also have the following directories.

  • docs – documentation files for the project that can be uploaded to and hosted on Read the Docs
  • plushcap – directory for source files

  • tests – location for files to test the package 

Next our new project will need its own virtualenv.

$ virtualenv --no-site-packages ~/Envs/plushcap
$ source ~/Envs/plushcap/bin/activate

Once the virtualenv is activated we’ll have a prompt like the following to let us know the virtualenv that is currently in use.

(plushcap)$

Initial Git commit

For the remainder of this tutorial assume that we’ve activated the plushcap virtualenv even if it is not shown at the prompt. Next we change into the Plushcap project directory.

$ cd ~/devel/py/plushcap

Let’s initialize the git repository and prep a push to a remote repository we’ll set up on GitHub.

Note that on Windows Git may add a remote repository after the git init command. Remove the incorrect remote repository with git remote remove origin then execute steps 2-4.

$ git init
$ git remote add origin git@github.com:makaimc/plushcap.git
$ git add -A .
$ git commit -m "Initial commit with skeleton project structure."

Let’s get our public repository set up on GitHub so we can push it there.

Selecting “New repository” will send us to a form where we fill in the repository details.

We now have an empty repository after filling in the details and pressing the green “Create repository” button.

Let’s push the local repository to our remote repository.

$ git push -u origin master

Now we’ll see the following code in the GitHub remote repository when we refresh the browser window.

We now have the skeleton of a Python package we can now fill our code into. You can see the results at this step or start from here with the tutorial-step-1 tag.

Writing the monitoring code

Let’s write some code to monitor if a website at an arbitrary URL is up or down. We’ll use the Requests library to easily retrieve the HTTP status code and content from URLs we want to monitor. We can then wrap conditional logic around responses to complete a simple version of our Plushcap library.

Requests can be installed through pip.

$ pip install requests

We also need to make sure Requests is in our requirements.txt file. We can freeze our current requirements with the freeze command.

$ pip freeze > requirements.txt

Let’s take a look at what’s in there so far. This list is from Linux so if you’re running through these commands on Windows or Mac OS X you may have more or less packages listed. As long as Requests is in there you’re in good shape.

$ more requirements.txt
argparse==1.2.1
requests==2.3.0
wheel==0.23.0

Wheel was installed by the cookiecutter package. Requests relies on argparse so that’s included as a dependency as well.

Let’s add some code to our library so it’s not just a skeleton structure package. Open a file named plushcap.py under the plushcap/ directory and add the following code into it.

import sys
import requests

CONNECTION_ERROR = -100

responses = {CONNECTION_ERROR: "is down or did not respond to the request.",
             200: "is online and returning 200 OK.",
             403: "is online but is denying the request due to lack of " + \
                  "permission.",
             404: "is online but the webpage at that URL was not found.",
             500: "is online but returning an internal server error.",
}

def contact_url(url):
    """
        Attempts to access the URL specified as a parameter. Returns the
        status code and the content for the request once it is complete.
    """
    try:
        response = requests.get(url)
        return response.status_code, response.content
    except requests.exceptions.ConnectionError:
        return CONNECTION_ERROR, ""

The contact_url function works but let’s leave it deliberately simple for this first iteration. We’ll add additional handling and functionality as we go along.

 

Let’s also add a main function so we can execute Plushcap from the command line.

 

if __name__=='__main__':
    if len(sys.argv) < 2:
        print("usage: python plushcap.py http://test.url/")
    else:
        status_code, content = contact_url(url=sys.argv[1])
        if responses.has_key(status_code):
            print(("The server at %s " + responses[status_code]) % sys.argv[1])
        else:
            print("Server response was unknown, status code: " + status_code)

We can now run the code from within the plushcap project subdirectory. The code will reach out to the URL specified as an argument and find out the HTTP status code. Based on the status code a message will be displayed whether the website is up, down or erroring out.

For example, try out the following command.

python plushcap.py http://www.fullstackpython.com/

If the server for http://www.fullstackpython.com/ is up then you’ll see the following success message.

The server at http://www.fullstackpython.com/ is online and returning 200 OK.

In case http://www.fullstackpython.com/ is down, you’ll instead see a message like this one.

The server at http://www.fullstackpython.com/ is down or did not respond to the request.

 

You can view the code we’ve written so far in the tutorial-step-2 tag on GitHub.

Testing functionality

Next we’ll add a few tests to ensure a basic scenarios work in this package. The tests will be run before every check in to ensure the main URL retrieval didn’t break with our changes. We’ll add more extensive tests and check our code coverage in a later step.

For now, we have two tests. The first test retrieves a known working URL for Full Stack Python. The second test attempts to contact a localhost server at a port which is unavailable. Add these to the tests/test_plushcap.py file.

 

    import sys
    import os

    import unittest

    sys.path.append(os.path.join('.', 'plushcap'))
    sys.path.append(os.path.join('..', 'plushcap'))
    from plushcap import plushcap

class TestPlushcap(unittest.TestCase):
    """
        Tests that URLs can be retrieved when they exist. URLs that do not
        exist or are down return the proper status codes.
    """
    def setUp(self):
        self.working_url = "http://www.fullstackpython.com/"
        self.non_existent_url = "http://localhost:8889/"

    def tearDown(self):
        pass

    def test_working_url(self):
        status_code, content = plushcap.contact_url(self.working_url)
        self.assertEquals(status_code, 200)

    def test_non_existent_url(self):
        status_code, content = plushcap.contact_url(self.non_existent_url)
        self.assertEquals(status_code, plushcap.CONNECTION_ERROR)

if __name__ == '__main__':
    unittest.main()

The tutorial-step-2 tag contains all the code to this point in the blog post.

Adding notifications

At this point we have a concise Python package that reaches out to a URL and prints a message based on the HTTP status code received. However, we’d like to monitor the URL on a regular interval and know if something happens to the website at the monitored URL. This step is where we can flesh out the code and integrate Twilio’s API to send SMS alerts when the website does not respond or returns an HTTP error status code.

Let’s add a new function to plushcap/plushcap.py that checks a URL at a regular frequency then sends an alert if the status code returned is not 200.

def check_url(url, twilio_client, twilio_from_number, alert_number,
    frequency=60):
    """
        Loops in a separate thread checking a given URL. Sends a 
        notification if the website goes down, 500 errors or gives a 404.
    """
    while True:
        status_code, content = contact_url(url)
        if status_code != 200:
            # alert time
            send_alert(url, status_code, twilio_client, twilio_from_number,
                alert_number)
    time.sleep(frequency)

Now let’s create another function that sets up the Twilio client and is the entry point when you set the Twilio environment variables for the Account SID, auth token, from number and alert number.

At the top of plushcap/plushcap.py add the following import.

from twilio.rest import TwilioRestClient

Then add the following function to plushcap/plushcap.py.

def monitor(url, frequency=60, twilio_account_sid=None, twilio_auth_token=None,
            twilio_from_number=None, alert_number=None):
    """
        Monitors the given URL. Default frequency is once a minute. Twilio 
        account SID and auth token are required for notifications and will be 
        pulled from the environment variables TWILIO_ACCOUNT_SID and 
        TWILIO_AUTH_TOKEN if not specified.
    """
    if not twilio_account_sid or not twilio_auth_token:
        # initialize with environment vars instead of function parameters
        twilio_client = TwilioRestClient()
    else:
        twilio_client = TwilioRestClient(twilio_account_sid, twilio_auth_token)
    if not twilio_from_number:
        twilio_from_number = get_env_setting('TWILIO_FROM_NUMBER')
    if not alert_number:
        alert_number = get_env_setting('ALERT_NUMBER')
    check_url(url, twilio_client, twilio_from_number, alert_number, frequency)

These values that need to be set for these environment variables are found on the dashboard of your account when you log in or in your account settings page. If you already have a Twilio account you can skip the following sign up steps. If not let’s quickly walk through the registration process.

First, sign up for a free Twilio account here.

After signing up Twilio needs to quickly verify not a malicious spam bot by either sending you a text message or giving you a voice call.

If you choose the text message verification you’ll receive a message like this one with a code to enter into the sign up form.

Twilio will assign a random phone number once the sign up process is complete. You can also search for another number through the Twilio web interface or purchase one programmatically through the API.

Answers to further phone number questions can be found on the phone number FAQ page.

Now we just need to grab the credentials from the account dashboard and export them as environment variables on the system that will run Plushcap.

Let’s modify the main function a bit to take these environment variables into account.

if __name__=='__main__':
    if len(sys.argv) < 2:
        print("usage: python plushcap.py http://test.url/")
        print("also ensure that TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, " + \
              "TWILIO_FROM_NUMBER\nand ALERT_NUMBER are set as " + \
              "environment variables")
    else:
        url = sys.argv[1]
        print "Monitoring %s" % url
        status_code, content = monitor(url)
        if responses.has_key(status_code):
            print(("The server at %s " + responses[status_code]) % url)
        else:
            print("Alert: status code received from %s is %i instead of " + \
                  "200 OK." % (sys.argv[1], status_code))

The tutorial-step-3 tag contains all the code we’ve written so far in this blog post.

Packaging and Uploading to PyPI

Now that we have the initial functionality for the package along with tests and documentation we can upload the library to PyPI. We need a MANIFEST.in file that tells setup.py what to include when we build a distributable package. Here’s our MANIFEST.in for Plushcap that sits in the base directory of the package:

include AUTHORS.rst
include CONTRIBUTING.rst
include HISTORY.rst
include LICENSE
include README.rst
include setup.py

recursive-include tests *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]

recursive-include docs *.rst conf.py Makefile make.bat

We’re including reStructedText (.rst) files, tests, our .py source files (but not the .pyc files) and the docs.

With this in place we can package up our Plushcap project and upload it to the central PyPI hosted packages repository. These commands should be run in the setup.py directory which is the base directory of our project. Note that your package cannot be named the same as an existing package in PyPI. Use PyPI’s search feature to find out what package names are available for new libraries. Once you’ve found an available name edit the package_name value in your project’s setup.py file. Then upload the new package with the following command.

$ python setup.py sdist upload

If everything goes well with the upload we’ll see output with a Server response (200): OK like the following screenshot (installation output capped for brevity).

Now Plushcap is on PyPI and can be installed by anyone with an Internet connection and pip installed by running the following command…

$ pip install plushcap

…and it’s installed!

Next up: Building a web application with Plushcap

The initial Plushcap library version is ready to go. But what if you want to use the code through a web application instead of directly through the Python interpreter? In the next post of this series we’ll take a look at using the a Python web application framework to build an application around the Plushcap library.

 

Authors
Sign up and start building
Not ready yet? Talk to an expert.