Airflow is an open source platform used to orchestrate workflows. It is perfect for Extract, Transform, Load tasks, data migration and data integration, among other jobs.

In this tutorial we will see how we can leverage Twilio’s Programmable Messaging to set up an alerting system for Airflow jobs.

Airflow DAG (source: Apache Airflow)

Tutorial requirements

To follow this tutorial you need to have:

• Git. See installation instructions here.
• Docker. You can follow the installation steps provided here.
• A Twilio account. If you are new to Twilio create a free account now. You can review the features and limitations of a free Twilio account.

Set up Airflow to run locally

We are going to start off by cloning a repository that contains the docker image we shall use for this tutorial. The image is based on Python 3.7-slim-buster and will create Postgres and Redis containers. Postgres is …