Mask Sensitive Data in Logs from the Twilio Python Helper Library

Mask Sensitive Data in Logs from the Twilio Python Helper Library

The Twilio Helper Library for Python has a very useful logging feature intended to help you debug issues. If you need to capture Twilio logs on the production version of your application, you may not want to include data that is considered personally identifiable information (PII) such as phone numbers or account identifiers. These details may not only be unnecessary for your debugging needs, but can also create a legal issue for your company in terms of compliance with privacy laws such as the GDPR.

In this article you will learn about two ways to eliminate PII from your log files.

Requirements

To work on this tutorial you will need the following items:

• Python 3.7 or newer. If your operating system does not provide a Python interpreter, you can go to python.org to download an installer.
• A Twilio account. If you are new to Twilio click here to create a free account.
• A Twilio phone number.
• A phone with active SMS service, for testing.

Setup a Python environment

To get started, open a terminal window and navigate to the place where you would like to set up your project.

Create a new directory called twilio-logging where your project will live, and change into that directory using the following commands:

Create a virtual environment

Following Python best practices, you are now going to create a virtual environment, where you are going to install the Python dependencies needed for this project.

If you are using a Unix or Mac OS system, open a terminal and enter the following commands to create and activate your virtual environment:

If you are following the tutorial on Windows, enter the following commands in a command prompt window:

Now you are ready to install the Twilio Helper Library for Python:

Define your Twilio credentials

To be able to access the Twilio service, the Python application will need to authenticate against Twilio servers with your Twilio account credentials. The most secure way to define these credentials is to add them as environment variables.

The information that you need is the “Account SID” and the “Auth Token”. You can find both on the main dashboard of the Twilio Console:

In your terminal, define two environment variables called TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN and set them to your account credentials:

If you are following this tutorial on a Windows computer, use set instead of export to define your environment variables in the command prompt.

If you want to learn more about other ways to set environment variables for Python applications, check out the Working with Environment Variables in Python tutorial.

How to log with the Twilio Helper Library for Python

The following snippet of Python code sends an SMS message. It is taken directly from the Programmable SMS Python Quickstart documentation.

Paste this code into your code editor, making sure to replace the placeholder phone numbers with your own. Save the file as send_sms.py on the project directory and run it as follows:

The script should run for a second or two, then print an identifier that starts with the letters SM and then exit. Shortly after, you should receive a test SMS on the phone number that you entered for the to argument.

The version of the script as presented above works silently, with the only output being the SMS identifier, which is explicitly printed in the bottom line.

When logging is configured, the Twilio helper library prints a lot of information that can be useful when debugging issues. The library documentation shows how to enable logging. Below is a version of the above script enhanced with logging, with the lines that were added are highlighted.

Run the script a second time to see all the information that is now printed to the terminal:

The information that is logged will largely depend on the version of the twilio library that you are using. Recent releases have gotten much better at omitting most PII. In the above example, the only piece of information that you may want to keep out of your logs is the Twilio Account SID that was used to send the message, which is shown as part of the request URL. Older versions of the library logged more information, including phone numbers, so your own application logs may show more information than the above example.

The first and simplest solution to avoid having PII in your logs is to upgrade to the most recent twilio library. You can do it with this command:

Add a logging filter

If after doing this you still find information written to the logs that you’d like to omit, then this section shows you how to add a logging filter that masks content that matches one or more regular expressions.

The trick is to create a custom formatter that in addition to doing formatting scans the lines of logging for patterns to mask out. The new version of the send_sms.py application with this capability is shown below:

Try to run the above version of the program to see the masking in effect. The line that contains the Twilio Account SID will now appear as follows:

This solution has some customization options. First of all, the patterns to mask and the replacement string are given in the filters array. For example, if you are using an old version of the Twilio library that logs phone numbers, you can add a second pattern for them:

The fmt argument that is passed to the SensitiveDataFormatter class can be used to configure the format of the log. The Python documentation lists all the logging attributes that are available. As an example, the following format adds a timestamp before all the other components:

With this change logged lines will look as follows:

Conclusion

In this article you have learned two ways of removing potentially sensitive information from your Twilio logs. I hope these solutions help you better protect the privacy of your users!

Miguel Grinberg is a Principal Software Engineer for Technical Content at Twilio. Reach out to him at mgrinberg [at] twilio [dot] com if you have a cool project you’d like to share on this blog!