Storing into AWS S3

Overview

You can write your Video Recordings and Compositions to your own AWS (Amazon Web Services) S3 bucket, rather than Twilio's cloud. This guide explains how you can setup your own Twilio account or project to use this capability.

Note: Once you activate external S3 storage, Twilio will stop storing Programmable Video audio/video recordings into the Twilio cloud. It will be your responsibility to manage the security and lifecycle of your recorded content.

Use this feature when you need to meet compliance requirements that excludes reliance on third-party storage.

Contents

• Recordings and Compositions in AWS S3
• Preparing your AWS account
  • Step 1: Create an AWS S3 bucket and obtain its URL
  • Step 2: Create an IAM user and get its credentials
• Configuring your Twilio Account
  • Step 3: Configure Twilio with the AWS Credentials you created
  • Step 4: Configure Twilio to store into the S3 bucket
    • Storing Recordings into the S3 bucket
    • Storing Compositions into the S3 bucket
• Advanced topics
  • Uploading to buckets with Server-Side Encryption (SSE)
• Known limitations

Recordings and Compositions in AWS S3

Video Recordings and Video Compositions have separated S3 storage settings. This means that S3 storage can be activated independently on them. However, composing Recordings requires access to their media, which is not available if they are in an external S3 bucket. As a result:

If you store your Recordings in S3, you will not be able to compose them.

If you need to compose your Recordings, you must store them in Twilio's cloud. However, those Recordings are only needed temporarily: as soon as the Composition is created they can be deleted permanently and irrevocably using Twilio's Video Recordings API.

Preparing your AWS account

To configure external S3 storage, you will need:

• The AWS S3 Bucket URL: The URL for the AWS S3 bucket of your choice.
• The AWS Credentials: AWS Credentials (i.e. an Access Key ID and a Secret Access Key) for an AWS Identity and Access Management (IAM) user with write access to the bucket.

The rest of this section explains how to gather the items above. If you already know how to obtain the AWS S3 Bucket URL and the AWS Credentials, you can skip step 1 and step 2 and jump to the next section.

Step 1: Create an AWS S3 bucket and obtain its URL

Amazon Simple Storage Service (S3) is designed to let you store and retrieve data from anywhere on the web. In S3, objects are stored into “buckets”. Those can be seen as virtual folders where objects can be written, read and deleted.

Create an AWS S3 bucket.

First, create an S3 bucket. Use whatever bucket configuration that makes sense for your application; Twilio does not have any special bucket requirements. Remember to make note of the following, which you will need later:

• The bucket-name. This must be DNS compliant.
• The bucket-region. This is the AWS region where your S3 bucket is located.

Get the AWS S3 Bucket URL.

Next, get the URL for your S3 bucket. We recommend that you use the virtual-host-style URL based on the scheme https://bucket-name.s3-aws-region-code.amazonaws.com. Note that bucket-name is the name of your bucket and that you must replace aws-region-code with the AWS region code corresponding to your bucket-region. Check the AWS documentation for getting your aws-region-code.

After completing this step, you should have an AWS S3 Bucket URL like:

• https://my-new-bucket.s3-us-east-2.amazonaws.com

Step 2: Create an IAM user and get its credentials

IAM is Amazon's product for controlling access to your AWS services. An IAM user is a person or service that can access your AWS resources.

Create an IAM user using the AWS Console. You should make note of the following:

Set Programmatic access as the “Access type” for your IAM user

Select Attach existing policies directly and press the Create policy button to configure the user permissions:

After that, select Create policy, pick the JSON editor and create a policy document with write permissions. You can use the following JSON snippet as a template for the policy document. Note:

• Replace my_bucket_name at the bottom of the snippet with the actual bucket-name, as obtained in step 1 above.
• Replace the string /folder/for/storage/ with the specific path where you want Twilio to store your recordings within your bucket (note that / is a valid path). Don't forget the * wildcard at the end.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "UploadUserDenyEverything",
            "Effect": "Deny",
            "NotAction": "*",
            "Resource": "*"
        },
        {
            "Sid": "UploadUserAllowPutObject",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my_bucket_name/folder/for/storage/*"
            ]
        }
    ]
}

Now, come back to the original browser tab and press the Refresh button to see the policy you created. You can select it and complete the IAM user creation.

Get the IAM Access Key ID, Secret Access Key, and Path

• Once the IAM user is created, Amazon provides you its credentials. They include an Access Key ID and a Secret Access Key. Store them in a secure location for later use.
• You must also note down the path (e.g. /folder/for/storage/) where you provide Twilio write permissions.

Configuring your Twilio Account

Step 3: Configure Twilio with the AWS Credentials you created

Next, you need to add a new AWS Credential to your Twilio account. For this, go to the Twilio Console AWS Credentials page and press Create new AWS Credential.

On the popup that opens, specify the friendly name you wish. Then, provide the AWS Access Key ID and the AWS Access Secret Key that you obtained in step 2 above. Finally, press Create.

After that, a newly created Twilio AWS Credential is listed in the Credentials page. Write down the AWS Credential SID that has the form CRxx.

Step 4: Configure Twilio to store into the S3 bucket

Remark that when you activate this feature in either Recordings or Compositions, a small .txt test file will appear into your bucket. Twilio uses that file for double checking that the write permissions you provided are working. You can remove the file safely if you want.

Storing Recordings into the S3 bucket

You have two options to enable Recordings S3 storage:

Enabling S3 Recordings storage using the Twilio's console

• Open the Twilio's Console in your account or project.
• Navigate to Programmable Video > Recordings > Settings.
• Enable External S3 Buckets and specify the AWS Credential you created in step 3 as well as the AWS S3 Bucket URL you obtained in step 1.
• Save your settings.
• All recordings created thereafter will be stored into the specified S3 bucket.

Enabling S3 Recordings storage using the Recording Settings API

Check the Recording Settings API documentation for detailed information on how to enable programmatically external S3 storage for your recordings.

Storing Compositions into the S3 bucket

You have two options to enable Compositions S3 storage:

Enabling S3 Compositions storage using the Twilio's console

• Open the Twilio's Console in your account or project.
• Navigate to Programmable Video > Compositions > Settings.
• Enable External S3 Buckets and specify the AWS Credential you created in step 3 as well as the AWS S3 Bucket URL you obtained in step 1.
• Save your settings.
• All compositions created thereafter will be stored into the specified S3 bucket.

Enabling S3 Compositions storage using the Recording Settings API

Check the Composition Settings API documentation for detailed information on how to enable programmatically external S3 storage for your compositions.

Advanced topics

Uploading to buckets with Server-Side Encryption (SSE)

Amazon S3 buckets support SSE (Server-Side Encryption). When enabled, all data written to disk is encrypted at the object level.

If you want to use SSE buckets to store your Twilio Recordings and Compositions, you must be aware that the only option we support is
SSE-KMS (SSE with AWS KMS-Managed Keys).

In order to make SSE-KMS to work with Twilio, you must grant access to the KMS key in your policy document. The following template illustrates how to do it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "UploadUserDenyEverything",
            "Effect": "Deny",
            "NotAction": "*",
            "Resource": "*"
        },
        {
            "Sid": "UploadUserAllowPutObject",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my_bucket_name/folder/for/storage/*"
            ]
        },
        {
            "Sid": "AccessToKmsForEncryption",
            "Effect": "Allow",
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": [
                "arn:aws:kms:region:account-id:key/key-id"
            ]
        }
    ]
}

Remember that:

• The Resource ARN parameter in UploadUserAllowPutObject must be replaced with the target bucket name and path, as specified in step 2 above.
• The Resource ARN parameter in AccessToKmsForEncryption must be replaced with the actual KMS ARN using the syntax specified in the official AWS documentation.
• The KMS key and the bucket must be in the same region (source).

Known limitations

• Twilio Programmable Video has a limitation with bucket names that contain non US characters (e.g. ü, ç, é, etc.). Please, make sure that the bucket names you use contain only 7-bit ASCII characters.