Mock it Til’ You Make It: Test Django with mock and httpretty

Mock it Til’ You Make It: Test Django with mock and httpretty

In this tutorial, we’ll learn how to use Python’s mock and httpretty libraries to test the parts of a Django app that communicate with external services. These could be 3rd party APIs, web sites you scrape, or essentially any resource which you don’t control and is behind a network boundary. In addition to that, we’ll also take a brief look into Django’s RequestFactory class and learn how we can use it to test our Django views. 🕵

In order to illustrate how all these pieces fit together, we’ll write a simple Django app —  “Hacker News Hotline”. It will fetch today’s headlines from Hacker News and serve them behind a TwiML endpoint, so anyone calling our Twilio number can hear the latest headlines on Hacker News.

Right, sounds like a plan. Let’s do this!

Python Environment Setup

Before we get our hands dirty, there are a few software packages we need to install.

1) First, we’ll need Python 2.7. Unfortunately httpretty doesn’t have official support for Python 3.x, as of now 😢. You can follow steps outlined in this StackOverflow post to install version 2.7. If you already have Python installed, you can quickly check which version you have by typing the command below in your terminal. Any 2.x version will work to follow this tutorial.

2) I recommend using virtualenv to keep your development machine tidy. Virtualenv gives you a virtual Python environment where all project dependencies can be installed to.

We’ll also need pip. Pip is a package manager for Python. You can follow this tutorial to set up both virtualenv and pip.

3) Ngrok is a reverse proxy solution, or in simple terms, software which exposes your development server to the outside world. You can install it here.

Last but not least, you’ll also need a Twilio account and a Twilio number with voice capability to test the project. You can sign up for a free trial here.

Django Project Setup

If you have the above environment setup done, let’s move into setting up our Django project.

First, let’s create a directory called twilio-project, then activate virtualenv and install all the dependencies we’ll need; django, httpretty, mock and twilio.

After that let’s start a django project called twilio_voice and add an app called hackernews_calling to our project. We will also need to apply initial database migrations for our Django app.

Open up your terminal and let’s start hacking. 👩‍💻👨‍💻

Great, now we can start writing some code. 👌

Fetching Hacker News Top Stories with Python

We’ll begin with writing a module that fetches top headlines from the Hacker News API. Within the hackernews_calling directory, create a file called hackernews.py with the following code:

The above module will do the following:

1. Fetch topstories.json file from remote host
2. Read a slice of the top_story_ids list
3. Fetch individual story details
4. Return a list of headlines

Simple.

Mocking HTTP Requests with Python

So how do we go about testing this module without firing HTTP requests every time we run our tests? Enter httpretty. httpretty is a library which monkey patches Python’s core socket module. It’s perfect for mocking requests and responses with whichever request library you’re using!

By default, Django creates a tests.py where you put all your test cases in one place. I prefer working with a directory structure with multiple test files instead. It helps to keep your tests files granular and concise.

Within the twilio_voice/hackernews_calling directory, apply the following bash magic. 🎩 🐰

Let’s create a test module called test_hackernews.py within the tests directory.

Now let’s investigate our test module closely 🔬.

The first thing you’ll probably notice is the @httpretty.activate decorator that we wrapped around our test method. This decorator replaces the functionality of Python’s core socket module and restores it to its original definition once our test method finishes executing during runtime. This technique is also known as “monkey patching”. Pretty neat, right?

Using httpretty we can register URIs to be mocked and define default return values for testing.

Here we are mocking the /v0/topstories.json endpoint to return a list of numbers — “ids”, from 1 to 10.

It’s also possible to mock URIs using regular expressions with httpretty. We leverage this feature to mock fetching individual story details.

Httpretty also lets us investigate the last request made. We use this feature to verify that the last request made was to fetch the 5th story item’s details.

Let’s run it. Jump back to the project root directory and run the tests.

You should see that test run was successful. 🎉

TwiML: Talking the Twilio Talk

TwiML is the markup language used to orchestrate Twilio services. With TwiML, you can define how to respond to texts and calls received by your Twilio number. We will generate a TwiML document to narrate the Hacker News data we fetch. When someone calls our Twilio number, they’ll hear the top headlines of the day! 🗣️📱

TwiML is simply an XML document with Twilio specific grammar. For example, to make your Twilio number speak upon receiving a call, you could use the following XML:

To make things easier we can use the official Twilio Python module to generate TwiML syntax.

Let’s create a new Django view (endpoint) to narrate Hacker News headlines using TwiML. We’ll return top Hacker News story headlines in TwiML format. Our hackernews_calling/views.py should look like this:

To start serving the TwiML doc, we’ll need to register the new endpoint we’ve introduced so our Django app serves it. Change the hackernews_callings/urls.py module to the following:

Mocking It: Testing Django Views with Mock and RequestFactory

If we were to test the view we have just written, every test run would make HTTP requests as the view is relying on the HackerNews service to fetch the data.

We could use httpretty again to fake requests at the network level, but there is a better solution in this scenario: the mock library we installed earlier. Mock is also part of the Python standard library since v.3.3.

Let’s write our test module and investigate it afterwards. Create a module called test_headlines_view.py under the tests directory with the following content:

You probably notice the @mock.patch decorator. This decorator monkey patches the function available at the path provided. Similar to how httpretty works, the mocked function is restored to its original state after the test method executes. You should always use absolute paths when working with the mock module as relative paths won’t work!

The second argument provided to the decorator is the value to return when the function is called. You may also have noticed we are passing the get_headlines_patched_func parameter into our test function. This acts as a spy so that we can interrogate if the mocked function was called, how many times it was called, with what arguments it was called and so forth.

Now let’s look into how we use Django’s internal RequestFactory. This module lets us imitate a Django http request object which can be passed into a View class. Using this approach we can test our views without actually hitting any endpoints.

This approach is useful to test any sorting, filtering, pagination or authorization logic within your views. It’s also possible to bake additional request headers into the faked Request object, such as authorization headers for restricted views. In our case, we’re simply mocking a GET request with the headline URI. The status code and response returned is checked afterwards.

Once again, jump back to the project root and run tests again. Everything should still be groovy. 👌

Putting Our Hacker News App Together

Now that we have our Django app set-up and working, let’s make it work with a Twilio number. (If you haven’t already signed up for Twilio, get a Trial account now.)

We’ll be using Twilio’s “Managing an Incoming Call” flow:

After installing ngrok, start it on port 8000 so the app is publicly available:

Pay attention to the output of ngrok, copy the URL provided and add it to the ALLOWED_HOSTS list within settings.py.

Now we can run the django app:

After starting the Django app, go to Twilio Dashboard > Phone Numbers > Active Numbers and click on a twilio number to set up the webhook URL. Copy and paste the secure forwarding URL (https) provided by ngrok and append /headlines to it. The HTTP action should be set to GET.

Give your Twilio number a call to listen to today’s Hacker News headlines. Voila! 🎉 🙌

Wrap Up: Hacker News Headlines with Django and Twilio

Congratulations if you’ve made it this far – give yourself a pat on the back! We’ve covered how to use mock, httpretty and RequestFactory modules to easily test Django. You can use mock to replace function bodies at runtime and httpretty to mock http requests at the network level. Both modules leverage monkey patching so mocked functions are restored to their original definitions after the test run. Last but not the least, we have also used RequestFactory to mock requests to test our Django views. You can find the finished project on GitHub.

If you would like to learn more about when you should use mocks, I recommend reading Eric Elliot’s “Mocking is a code smell” post here.

Thanks for reading, please do let me know if you have any questions in the comments.

Ersel Aker is a full-stack developer based in the UK. He has been working with FinTech startups using Python, Nodejs and React. He is also the author of Spotify terminal client  library, and contributor to various open source projects. You can find him on Twitter, GitHub, Medium or at erselaker.com