How to Use a Task Queue in Python to Send Twilio SMS

Time to read: 5 minutes

August 31, 2020
Written by
Diane Phan
Twilion
Reviewed by
Miguel Grinberg
Twilion
Matt Makai
Twilion

header - How to Use a Task Queue in Python to Send Twilio SMS

It’s frustrating to deal with an API that enforces a rate limit, especially if you have a US or Canada phone number and can only send one SMS message per second. Fortunately as a developer, we can find a workaround solution so that you can continue to send out text messages timely and stick to your schedule without receiving 429 status codes.

In this tutorial we will implement a task queue in Python to send Twilio SMS messages in a timely manner.

gif of SMS text message demo alongside a timer

Tutorial Requirements

Let’s talk about task queues

Task queues are a great way to allow tasks to work asynchronously outside of a single request. This is incredibly helpful when managing heavy workloads that might not work efficiently when called all at once, or when making large numbers of calls to a database that return data slowly over time rather than all at once.

There are many task queues in Python to assist you in your project, however, we’ll be discussing a solution today known as RQ.

RQ, also known as Redis Queue, is a Python library that allows developers to queue jobs that are processed in the background. Using the connection to Redis, it’s no surprise that this library is super lightweight and offers support for those getting started for the first time.  

By using this particular task queue, RQ makes it possible to process jobs in the background with little to no hassle.

Set up the environment

Create a project directory in your terminal called “twilio-taskqueue” to follow along.

$ mkdir twilio-taskqueue
$ cd twilio-taskqueue

Install a virtual environment and copy and paste the commands to install rq and related packages. If you are using a Unix or MacOS system, enter the following commands:

$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install rq requests rq-scheduler python-dotenv pytz twilio

If you are on a Windows machine, enter the following commands in a prompt window:

$ python -m venv venv
$ source venv\bin\activate
(venv) $ pip install rq requests rq-scheduler python-dotenv pytz twilio

RQ requires a Redis installation on your machine which can be done using the following commands using wget. Redis is on version 6.0.6 at the time of this article publication.

If you are using a Unix or MacOS system, enter these commands to install Redis.

$ wget http://download.redis.io/releases/redis-6.0.6.tar.gz
$ tar xzf redis-6.0.6.tar.gz
$ cd redis-6.0.6
$ make

Run the Redis server in a separate terminal window on the default port with the command src/redis-server from the directory where it's installed.

For Windows users, you would have to follow a separate tutorial to run Redis on Windows. Download the latest zip file on GitHub and extract the contents.

Run the redis-server.exe file that was extracted from the zip file to start the Redis server.

The output should look similar to the following after running Redis:

93368:C 05 Aug 2020 21:37:01.865 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
93368:C 05 Aug 2020 21:37:01.865 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=93368, just started
93368:C 05 Aug 2020 21:37:01.865 # Warning: no config file specified, using the default config. In order to specify a config file use src/redis-server /path/to/redis.conf
93368:M 05 Aug 2020 21:37:01.866 * Increased maximum number of open files to 10032 (it was originally set to 2560).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.0.6 (00000000/0) 64 bit
  .-`` .-

Keep the Redis server running on a separate terminal tab as you follow along in this tutorial.

Authenticate against Twilio Service

We need to create a .env file to safely store some important credentials that will be used to authenticate against the Twilio service. We’re going to set this up now but we’ll discuss more about it in the next section.

Here is what the .env file should look like:  

TWILIO_ACCOUNT_SID="<your account SID>"
TWILIO_AUTH_TOKEN="<your auth token>"

For the TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN variables, you can obtain the values that apply to your Twilio account from the Twilio Console:

Account SID and Auth Token fields for Twilio Credentials

Set up a contact .csv file

You might already have a .csv file of your own, but for the purpose of this tutorial, I created a  .csv file with random information from the Web and replaced the phone numbers with my own. I also repeated the same contact three times to ensure that 3 text messages are sent to my phone number.

Here’s my sample that you can copy and save into your directory. Replace the phone numbers with your own.

first_name, last_name, company_name, address, city, county, state, zip, phone1, phone, email
Diane,Robinson,twilio,34 Center St,Hamilton,Butler,OH,45011,+<YOUR_PHONE_NUMBER>,+<YOUR_PHONE_NUMBER>,iluvporterrobinson@lol.net
Diane,Robinson,twilio,34 Center St,Hamilton,Butler,OH,45011,+<YOUR_PHONE_NUMBER>,+<YOUR_PHONE_NUMBER>,iluvporterrobinson@lol.net
Diane,Robinson,twilio,34 Center St,Hamilton,Butler,OH,45011,+<YOUR_PHONE_NUMBER>,+<YOUR_PHONE_NUMBER>,iluvporterrobinson@lol.net

Redis queue to the rescue

Open a new tab on the terminal so that Redis continues to run on the initial tab.

Create an app.py file in the root directory and add the following code:

from datetime import datetime, timedelta
import msg 
import pytz
import csv

from redis import Redis
from rq import Queue

queue = Queue(connection=Redis())

contacts_file = open('samplecontacts.csv')
csv_file = csv.reader(contacts_file)

def get_next_contact():
    next_scheduled_time = datetime.now()
    delta = 0
    for row in csv_file:
        recipient_number = row[8]        # the column for the phone number 
        queue.enqueue_in(timedelta(seconds=delta), msg.send_text_message,
                               <YOUR-TWILIO_NUMBER>, recipient_number,
                                'Hello this is a message')
        delta += 15

def main():
    get_next_contact()

if __name__ == "__main__":
    main()

A queue object is created here to keep track of functions to be executed.

The enqueue_in function expects a timedelta in order to schedule the specified job. The datetime module to specify the time difference between each text message sent. In this case, seconds is specified, but this variable can be changed according to the time schedule expected for your usage. Check out other ways to schedule a job on this GitHub README.

The delta variable is set to +=15 which means that every 15 seconds, a new text message will be sent out.

Create another file in the root directory called msg.py and paste the following code:

import os
from dotenv import load_dotenv
import requests
from twilio.rest import Client
load_dotenv()

client = Client()

def send_text_message(from_, to, body):
    client.messages.create(from_=from_, to=to, body=body)

This file was created in order to send the text messages to the contacts taken from your .csv file.

Run the redis queue

The full code for this tutorial can be found on my GitHub.

The Redis server should still be running in a tab from earlier in the tutorial at this point. If it stopped, run the command src/redis-server inside the redis-6.0.6 folder on one tab. Open another tab solely to run the RQ scheduler with the command rq worker --with-scheduler. For developers with a Windows machine, start the redis-cli.exe to run the command above.

This should be the output after running the command above.

09:05:05 Worker rq:worker:1c0a81e7e160458283ebe1e21d92a26e: started, version 1.5.0
09:05:05 *** Listening on default...
09:05:05 Cleaning registries for queue: default
INFO:rq.worker:Cleaning registries for queue: default

Lastly, open a third tab in the terminal for the root project directory. Start up the virtual environment again with the command source venv/bin/activate. Then type python app.py to run the project.

Have your phone next to you and wait delta time for the next message to appear.

In the gif demo below, notice that 3 text messages were sent, as there were 3 contacts in the .csv file from earlier. They were scheduled to be sent every 15 seconds and although the gif wasn’t able to capture the best timing due to having to run the program and record, it is noticeable that it took at least 45 seconds to send out the SMS text messages.

gif of SMS text message demo alongside a timer

Here’s the sample output inside of the rqworker tab. Notice that for every line with the phrase default: Job OK, the timestamp before the phrase shows a time that is 15 seconds before the next scheduled job.

11:31:58 default: msg.send_text_message(<YOUR-TWILIO_NUMBER>, +<YOUR_PHONE_NUMBER>, 'Hello this is a message') (cdb76ee2-7b7d-4bbb-886d-e74d4853daed)
INFO:rq.worker:default: msg.send_text_message(<YOUR-TWILIO_NUMBER>, +<YOUR_PHONE_NUMBER>, 'Hello this is a message') (cdb76ee2-7b7d-4bbb-886d-e74d4853daed)
11:31:59 default: Job OK (cdb76ee2-7b7d-4bbb-886d-e74d4853daed)
INFO:rq.worker:default: Job OK (cdb76ee2-7b7d-4bbb-886d-e74d4853daed)
11:31:59 Result is kept for 500 seconds
INFO:rq.worker:Result is kept for 500 seconds
11:32:13 default: msg.send_text_message(<YOUR-TWILIO_NUMBER>, +<YOUR_PHONE_NUMBER>, 'Hello this is a message') (49ede6b1-04c7-4797-b9f5-e8fa2c20759f)
INFO:rq.worker:default: msg.send_text_message(<YOUR-TWILIO_NUMBER>, +<YOUR_PHONE_NUMBER>, 'Hello this is a message') (49ede6b1-04c7-4797-b9f5-e8fa2c20759f)
11:32:14 default: Job OK (49ede6b1-04c7-4797-b9f5-e8fa2c20759f)
INFO:rq.worker:default: Job OK (49ede6b1-04c7-4797-b9f5-e8fa2c20759f)
11:32:14 Result is kept for 500 seconds
INFO:rq.worker:Result is kept for 500 seconds
11:32:28 default: msg.send_text_message(<YOUR-TWILIO_NUMBER>, +<YOUR_PHONE_NUMBER>, 'Hello this is a message') (4358593b-bb60-4e1e-8ee1-c8817663dd4d)
INFO:rq.worker:default: msg.send_text_message(<YOUR-TWILIO_NUMBER>, +<YOUR_PHONE_NUMBER>, 'Hello this is a message') (4358593b-bb60-4e1e-8ee1-c8817663dd4d)
11:32:29 default: Job OK (4358593b-bb60-4e1e-8ee1-c8817663dd4d)
INFO:rq.worker:default: Job OK (4358593b-bb60-4e1e-8ee1-c8817663dd4d)
11:32:29 Result is kept for 500 seconds

What’s next for task queues?

Congratulations! You have successfully implemented a task queue in Python using Redis Queue to send Twilio SMS messages in a timely manner. No more 429 status codes for us!

Interested in new projects that require task queues? Check these out:

Let me know what you have been building by reaching out to me over email!

Diane Phan is a Developer Network editor on the Developer Voices team. She loves to help programmers tackle difficult challenges that might prevent them from bringing their projects to life. She can be reached at dphan [at] twilio.com or LinkedIn.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.