By
The Twilio Helper Library for Python has a very useful logging feature intended to help you debug issues. If you need to capture Twilio logs on the production version of your application, you may not want to include data that is considered personally identifiable information (PII) such as phone numbers or account identifiers. These details may not only be unnecessary for your debugging needs, but can also create a legal issue for your company in terms of compliance with privacy laws such as the GDPR.
In this article you will learn about two ways to eliminate PII from your log files.
Requirements
To work on this tutorial you will need the following items:
- Python 3.7 or newer. If your operating system does not provide a Python interpreter, you can go to python.org to download an installer.
- A Twilio account. If you are new to Twilio click here to create a free account.
- A Twilio phone number.
- A phone with active SMS service, for testing.
Setup a Python environment
To get started, open a terminal window and navigate to the place where you would like to set up your project.
Create a new directory called twilio-logging where your project will live, and change into that directory using the following commands:
mkdir twilio-logging
cd twilio-logging
Create a virtual environment
Following Python best practices, you are now going to create a virtual environment, where you are going to install the Python dependencies needed for this project.
If you are using a Unix or Mac OS system, open a terminal and enter the following commands to create and activate your virtual environment:
python3 -m venv venv
source venv/bin/activate
If you are following the tutorial on Windows, enter the following commands in a command prompt window:
python -m venv venv
venv\Scripts\activate
Now you are ready to install the Twilio Helper Library for Python:
pip install twilio
Define your Twilio credentials
To be able to access the Twilio service, the Python application will need to authenticate against Twilio servers with your Twilio account credentials. The most secure way to define these credentials is to add them as environment variables.
The information that you need is the “Account SID” and the “Auth Token”. You can find both on the main dashboard of the Twilio Console:
In your terminal, define two environment variables called TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN and set them to your account credentials:
export TWILIO_ACCOUNT_SID=xxxxxxxxx
export TWILIO_AUTH_TOKEN=xxxxxxxxx
If you are following this tutorial on a Windows computer, use set instead of export to define your environment variables in the command prompt.
If you want to learn more about other ways to set environment variables for Python applications, check out the Working with Environment Variables in Python tutorial.
How to log with the Twilio Helper Library for Python
The following snippet of Python code sends an SMS message. It is taken directly from the Programmable SMS Python Quickstart documentation.
import os
from twilio.rest import Client
account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
client = Client(account_sid, auth_token)
message = client.messages.create(
body="Join Earth's mightiest heroes. Like Kevin Bacon.",
from_='+15017122661', # <-- write your Twilio number here
to='+15558675310' # <-- write your own phone number here
)
print(message.sid)
Paste this code into your code editor, making sure to replace the placeholder phone numbers with your own. Save the file as send_sms.py on the project directory and run it as follows:
python send_sms.py
SM957db77221f3b1b5b3aefd2a5d2b121c
The script should run for a second or two, then print an identifier that starts with the letters SM and then exit. Shortly after, you should receive a test SMS on the phone number that you entered for the to argument.
The version of the script as presented above works silently, with the only output being the SMS identifier, which is explicitly printed in the bottom line.
When logging is configured, the Twilio helper library prints a lot of information that can be useful when debugging issues. The library documentation shows how to enable logging. Below is a version of the above script enhanced with logging, with the lines that were added are highlighted.
import logging
import os
from twilio.rest import Client
account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
client = Client(account_sid, auth_token)
logging.basicConfig()
client.http_client.logger.setLevel(logging.INFO)
message = client.messages.create(
body="This is a test message",
from_='+15017122661', # <-- write your Twilio number here
to='+15558675310' # <-- write your own phone number here
)
print(message.sid)
Run the script a second time to see all the information that is now printed to the terminal:
INFO:twilio.http_client:-- BEGIN Twilio API Request --
INFO:twilio.http_client:POST Request: https://api.twilio.com/2010-04-01/Accounts/AC1111111111111111111111111111111/Messages.json
INFO:twilio.http_client:Headers:
INFO:twilio.http_client:User-Agent : twilio-python/7.16.2 (Darwin x86_64) Python/3.11.0
INFO:twilio.http_client:X-Twilio-Client : python-7.16.2
INFO:twilio.http_client:Accept-Charset : utf-8
INFO:twilio.http_client:Content-Type : application/x-www-form-urlencoded
INFO:twilio.http_client:Accept : application/json
INFO:twilio.http_client:-- END Twilio API Request --
INFO:twilio.http_client:Response Status Code: 201
INFO:twilio.http_client:Response Headers: {'Date': 'Fri, 03 Feb 2023 15:58:24 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Content-Length': '786', 'Connection': 'keep-alive', 'Twilio-Request-Id': 'RQ1d6ba900274b1bbebab03b8332268969', 'Twilio-Request-Duration': '0.139', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'Accept, Authorization, Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since, Idempotency-Key', 'Access-Control-Allow-Methods': 'GET, POST, DELETE, OPTIONS', 'Access-Control-Expose-Headers': 'ETag', 'Access-Control-Allow-Credentials': 'true', 'X-Powered-By': 'AT-5000', 'Twilio-Concurrent-Requests': '1', 'X-Shenanigans': 'none', 'X-Home-Region': 'us1', 'X-API-Domain': 'api.twilio.com', 'Strict-Transport-Security': 'max-age=31536000'}
SM1d6ba900274b1bbebab03b8332268969
The information that is logged will largely depend on the version of the twilio library that you are using. Recent releases have gotten much better at omitting most PII. In the above example, the only piece of information that you may want to keep out of your logs is the Twilio Account SID that was used to send the message, which is shown as part of the request URL. Older versions of the library logged more information, including phone numbers, so your own application logs may show more information than the above example.
The first and simplest solution to avoid having PII in your logs is to upgrade to the most recent twilio library. You can do it with this command:
pip install –upgrade twilio
Add a logging filter
If after doing this you still find information written to the logs that you’d like to omit, then this section shows you how to add a logging filter that masks content that matches one or more regular expressions.
The trick is to create a custom formatter that in addition to doing formatting scans the lines of logging for patterns to mask out. The new version of the send_sms.py application with this capability is shown below:
import logging
import os
import re
from twilio.rest import Client
account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
client = Client(account_sid, auth_token)
class SensitiveDataFormatter(logging.Formatter):
@staticmethod
def _filter(s):
filters = [
[r'(AC[a-f0-9]+)', '[twilio account redacted]'],
]
for f in filters:
s = re.sub(f[0], f[1], s)
return s
def format(self, record):
original = logging.Formatter.format(self, record)
return self._filter(original)
handler = logging.StreamHandler()
handler.setFormatter(SensitiveDataFormatter(fmt='%(levelname)s:%(name)s:%(message)s'))
client.http_client.logger.addHandler(handler)
client.http_client.logger.setLevel(logging.INFO)
message = client.messages.create(
body="This is a test message",
from_='+15017122661', # <-- write your Twilio number here
to='+15558675310' # <-- write your own phone number here
)
print(message.sid)
Try to run the above version of the program to see the masking in effect. The line that contains the Twilio Account SID will now appear as follows:
INFO:twilio.http_client:POST Request: https://api.twilio.com/2010-04-01/Accounts/[twilio account redacted]/Messages.json
This solution has some customization options. First of all, the patterns to mask and the replacement string are given in the filters array. For example, if you are using an old version of the Twilio library that logs phone numbers, you can add a second pattern for them:
filters = [
[r'(AC[a-f0-9]+)', '[twilio account redacted]'],
[r'(\+[0-9]+)', '[phone_redacted]'],
]
The fmt argument that is passed to the SensitiveDataFormatter class can be used to configure the format of the log. The Python documentation lists all the logging attributes that are available. As an example, the following format adds a timestamp before all the other components:
handler.setFormatter(SensitiveDataFormatter(fmt='%(asctime)s:%(levelname)s:%(name)s:%(message)s'))
With this change logged lines will look as follows:
2023-02-03 17:34:52,919:INFO:twilio.http_client:POST Request: https://api.twilio.com/2010-04-01/Accounts/[twilio account redacted]/Messages.json
Conclusion
In this article you have learned two ways of removing potentially sensitive information from your Twilio logs. I hope these solutions help you better protect the privacy of your users!
Miguel Grinberg is a Principal Software Engineer for Technical Content at Twilio. Reach out to him at mgrinberg [at] twilio [dot] com if you have a cool project you’d like to share on this blog!
