Customer Security Notice on CVE-2014-0160 / Heartbleed Disclosure

Customer Security Notice on CVE-2014-0160 / Heartbleed Disclosure

The engineering team at Twilio has been working to assess the impact for our customers in the wake of April 7th’s disclosure of CVE-2014-0160, known colloquially as Heartbleed. We join nearly every service provider on the Internet responding to this critical vulnerability in OpenSSL’s handling of heartbeat packets and conducted a comprehensive security review in response.  Our obligation as a custodian of your communication compels a unique urgency with disclosures such as these – here’s what we know, what you need to do and where you can find additional help from us.

Audit Results

Twilio’s public services are not secured by a version of OpenSSL with the Heartbleed vulnerability.  The Twilio API, account portal and interoperations with our network service providers were checked and tested for vulnerable versions of OpenSSL.  Further, although Twilio utilizes Amazon Web Services, our service is not served through products that were discovered to be vulnerable.

If the hosting provider or OpenSSL deployment for your Twilio application was not affected, there is nothing further you need to do.  However, many of our customers host their Twilio applications with hosting providers that were affected by this vulnerability.

All customers are strongly urged to audit the service providers hosting their Twilio applications to determine if their Twilio credentials may have been exposed.

Two third-party providers that provide ancillary services to Twilio were found to be affected. The vulnerabilities were reported following each provider’s responsible disclosure procedure.

The transactional email provider that delivers password reset requests was discovered to be vulnerable at 11:31am PDT and reported shortly thereafter.  The provider confirmed their OpenSSL deployment was patched and their SSL certificates were reissued at 3:44pm PDT.

Additionally, the translation service that hosts the Japanese version of the twilio.com website (http://jp.twilio.com) was discovered to be vulnerable at 3:35pm PDT. The hosting provider confirmed their OpenSSL deployment was patched at 4:28pm PDT.  Certificates for jp.twilio.com were reissued by the Twilio operations team and installed at 6:52pm PDT.

After our review of the account activity for affected accounts there is no evidence that any Twilio users were compromised. However, given the nature of this vulnerability, we’re urging our customers who have reset their passwords recently or logged in via jp.twilio.com to take some additional steps to ensure the security of their Twilio applications. Those customers should follow the Auth Token reset and password change steps outlined below.

While Twilio services appear to be unaffected, we recognize a number of you may be using hosting providers or OpenSSL deployments that may be.  Here is a quick walkthrough of how to determine if your host is affected.

How To Determine If Your Hosting Provider Is Affected By Heartbleed

You can determine if your hosting provider is affected by following these steps.

Step 1: Check Your Twilio Endpoints

Customers are encouraged to check all hosts serving as endpoints for Voice and Messaging Request URLs.  You can find these by looking at your Twilio Phone Numbers and TwiML Applications.  For example, you can find your Voice and Messaging Request URLs for a phone number by visiting the configuration for that phone number:

With the hosts identified, next step is to check for the Heartbleed vulnerability.  This can be done two ways.  First, one can verify the version of OpenSSL running on the host using the openssl utility:

openssl version

From Codenomicon’s disclosure of the vulnerability, the version number should be compared to the following rubric:

• OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable
• OpenSSL 1.0.1g is NOT vulnerable
• OpenSSL 1.0.0 branch is NOT vulnerable
• OpenSSL 0.9.8 branch is NOT vulnerable

Additionally, you can also check if your host is vulnerable using a tool testing the heartbeat extensions of your server.  A number of tools have emerged to test for the Heartbleed vulnerability.  Here are a few we found:

• Filippo Valsorda’s Heartbleed Test
• 1st Limited’s Heartbleed Test
• Jared Stafford’s Proof of Concept exploit in Python

Step 2: Check Your Hosts Holding Twilio Account Credentials

If you have hosts exposed to the public Internet that store your Twilio Account Sid and Auth Token, check these hosts using the methods described above.  This vulnerability allows remote reading of the memory of the host.  Affected hosts could then leak Twilio account credentials that are stored in memory.

If Your Hosting Provider Is Affected, How To Secure The Vulnerability

Step 1: Repair Any Vulnerable Systems

Customers with vulnerable versions of OpenSSL are encouraged to repair their deployments one of two ways.

Customers can upgrade to a fixed version of OpenSSL.  This can be achieved either by using their operating system’s package manager (such as apt, yum or up2date) or by downloading and installing the latest release from OpenSSL.

Alternatively, customers can recompile OpenSSL with the following flag to disable the affected heartbeat extensions: `-DOPENSSL_NO_HEARTBEATS`.

Step 2: If Necessary, Reset Your Twilio Auth Token

While no evidence indicates that any customer accounts were compromised as the result of this vulnerability, customers with vulnerable hosts are encouraged to reset their Auth Tokens out of an abundance of caution.

Note: an Auth Token reset will require you to update your Twilio production applications. This is not necessary if your hosts were not affected.

You can reset your Auth Token by visiting your Account Settings:

Again, resetting your Auth Token will require you to update your Twilio applications to prevent service interruption.  Resetting an Auth Token will cause Twilio applications configured for your previous credentials to fail.  Coordinating this reset and the configuration of your application must be done to avoid a service outage.

If you have any question about whether or not resetting your Auth Token is prudent, please reach out to our Customer Support team at help@twilio.com.

Step 3: If using jp.twilio.com, Change Your Twilio Password

Though we have no evidence to suggest exploitation of our Japanese translated site as a result of the affected hosting provider, we do recommend a further action for these customers for greater assurance.  In addition to resetting your Auth Token, customers who sign in via http://jp.twilio.com are encouraged to change their passwords.

You can reset your Twilio password by visiting your User Settings and scrolling down until you reach the Change Password form:

Further Resources for Heartbleed Help

Everyone deploying production services on the Internet is working to mitigate the effects of this vulnerability.  Here is a quick roundup of resources we found useful in our response to this disclosure:

• Codenomicon’s original disclosure with FAQ
• NIST’s National Vulnerability Database entry
• Lifehacker’s Heartbleed breakdown
• Mustafa Al-Bassam’s test of Top 1000 sites.

As always, if you have any questions about the security of your Twilio account, contact us at help@twilio.com.  If you are a security professional who has discovered any vulnerability in the Twilio service, we warmly welcome your responsible disclosure.

We’ll continue to monitor this issue as the community and vendors investigate this vulnerability further.

 Update: 11 April 2014 8:36am PDT – As a precautionary step for our customers, this morning we issued new SSL certificates for twilio.com.  No additional action is required by our customers nor does it represent any new information we have gathered around Heartbleed.  Our OpenSSL deployment remains unaffected by the vulnerability, but we believe joining the rest of the industry in this precaution provides additional assurance to our customers.

Update #2: 17 June 5:38 PDT – Version 1.1.4 of our Twilio Client iOS SDK, originally made available on 17 April 2014, eliminated a version of the OpenSSL library affected by Heartbleed, replaced it with an unaffected version and disabled SSL heartbeats. As a client-side library, the SDK’s use of OpenSSL only posed a risk in a sophisticated man-in-the-middle attack scenario and would have been difficult to exploit. In line with our responsible disclosures, we privately notified all customers who had recently used the SDK of the vulnerability on 17 April 2014, providing them time to release updated versions of their application to their app marketplaces. We have no evidence of exploits against older versions of the SDK. Please contact Twilio Support at help@twilio.com with any questions.

For further questions, please reach out to help@twilio.com.