Working with VP8 Simulcast

Contents

• What is simulcast
  • How Twilio uses simulcast
  • Pros and cons of simulcast
  • Adaptive simulcast
  • Limitations
• Resolution and simulcast layers
• Enable simulcast in your Twilio application
  • Enable adaptive simulcast using the JavaScript SDK
  • Enable standard simulcast using the JavaScript SDK
  • Enable simulcast using the iOS SDK
  • Enable simulcast using the Android SDK
• Simulcast and capture settings on mobile SDKs

What is simulcast

Simulcast is a standardized technique used to retain media quality when some subscribers have limited bandwidth. It is a mechanism for providing scalability to non-scalable video codecs such as VP8.

With simulcast, a client sends multiple versions of the same video simultaneously. Each version is encoded independently at a different resolution and frame rate; this way, a subscriber with limited bandwidth can receive a lower quality version of the media, but subscribers with more bandwidth can receive a higher quality video and their media quality is not degraded.

How Twilio uses simulcast

Twilio Video provides the option to use VP8 simulcast in Group Rooms via Twilio's Selective Forwarding Unit (SFU). To learn more about SFUs and media exchange in Peer-to-Peer vs. Group Rooms, see Understanding Video Rooms.

SFUs simply forward media and can neither transcode nor modify the video. When sending media with unicast (only sending one version of the video track), publishers need to reduce quality to adapt to the worst of their subscribers' bandwidths so that no subscriber is congested.

With VP8 simulcast, the SFU can forward higher quality videos to higher bandwidth subscribers and lower quality videos to lower bandwidth ones. The track publisher sends different track qualities and the SFU selects the most optimal quality for each subscriber. Then, the subscriber receives a single VP8 encoded video that is most suited for their network conditions.

The following illustration shows the difference between unicast and simulcast in a Group Room.

The SFU is critical to enabling simulcast. Simulcast is not recommended in Peer-to-Peer rooms, because there is no SFU mediating the data being sent. Simulcast in a Peer-to-Peer room means that each client is sending multiple differently encoded versions to every other participant in the room, which will consume more resources for both senders and receivers.

Pros and cons of simulcast

Simulcast offers the following benefits for Group Room participants:

• VP8 subscribers can receive video adapted to their available bandwidth. This significantly improves the quality on Group Rooms with many heterogeneous Participants.
• VP8 subscribers are isolated from each other so that a subscriber with a degraded network link does not affect the reception quality of other subscribers.

On the other hand, simulcast also has some drawbacks:

• Simulcast only contributes to improve the video quality in Group Rooms with three or more Participants.
• On mobile devices, publishers' battery consumption is higher because the publisher encodes multiple versions of the same video track.
• Publishers' bandwidth consumption is higher (up to double in some cases) because the publisher sends multiple versions of the same video track. This increase does not impact your Programmable Video costs as Twilio does not charge for upstream bandwidth (i.e. from the sender to Twilio’s cloud).

Adaptive simulcast

Twilio's standard VP8 simulcast sends up to three layers of video at different resolutions. See the approximate resolutions of each layer here. In some video conferencing contexts, the higher resolution layers that consume the most resources to encode may not be needed.

Twilio Video offers adaptive simulcast, which enables and disables simulcast layers dynamically to improve bandwidth and CPU usage. This helps save device resources in cases such as presentation and grid modes, when the application does not need a Participant's highest resolution video. Adaptive simulcast ensures that publishers are only encoding the spatial layers needed at a given moment.

For example, when someone is presenting in a video conference, you will frequently only display the presenter's video in a large format, and will display only thumbnails of other participants' video. The participants who are not presenting do not need to encode and send higher resolution video layers because their video is not highlighted. The same might also be true in a video conference in grid mode, where each participant's video is the same size and no one's video needs to be the highest quality.

In these situations, adaptive simulcast can detect which layers are being used by subscribers and automatically turn off encoding on the publisher's side for higher spatial layers that are not being used. As speakers change, adaptive simulcast will dynamically turn on or turn off the appropriate spatial layers, based on what subscribers in the room are using.

Note that adaptive simulcast will not disable any video layers when the Room is being recorded, to help produce high quality recordings.

Adaptive simulcast is currently only available in the Twilio Video JavaScript SDK. See Enable adaptive simulcast using the JavaScript SDK for more information. If your application is currently using VP8 simulcast with the JavaScript SDK, we recommend that you switch to this option.

Limitations

• Simulcast should only be used in Group Rooms. Using it in Peer-to-Peer Rooms does not improve quality and degrades application performance.
• Simulcast is only supported for the VP8 video codec.

Resolution and simulcast layers

Twilio SDKs encode up to three spatial layers when simulcast is enabled. The following table illustrates which layers are typically generated given a particular capture resolution. Remark that this is just an approximation and that the real behavior may be slightly different. In the table, disabled means that that layer is not sent in those conditions. (Video of the specified resolution is not generated by the publisher and is not available at the SFU to be forwarded to subscribers).

Enable simulcast in your Twilio application

Simulcast can be enabled in Group Rooms. The following table illustrates Twilio’s current support for simulcast:

Enable adaptive simulcast using the JavaScript SDK

Simulcast is disabled by default. You can enable simulcast on a per-Participant basis when connecting to a Room.

To enable adaptive simulcast, set preferredVideoCodecs="auto" in ConnectOptions when connecting to a video Room. The SDK will use VP8 simulcast, and will enable/disable simulcast layers dynamically, thus improving bandwidth and CPU usage.

Adaptive simulcast works best when used along with Client Track Switch Off Control and Video Content Preferences. These two flags allow the SFU to determine which simulcast layers are needed, thus allowing it to disable the layers not needed on publisher side.

const { connect } = require('twilio-video');

const room = await connect(token, {
  preferredVideoCodecs: 'auto',
  bandwidthProfile: {
    video: {
      contentPreferencesMode: 'auto',
      clientTrackSwitchOffControl: 'auto'
    }
  }
});

Please note the following limitations with adaptive simulcast in the JavaScript SDK:

• Specifying preferredVideoCodecs="auto" will revert to unicast in the following cases:
  • The publisher is using Firefox
  • The publisher has preferred the H264 codec
  • The Room is configured to support only the H264 codec
  • Peer-to-Peer Rooms
• When the Room is being recorded, the SFU will not disable any simulcast layers of the publisher's VideoTrack.

Enable standard simulcast using the JavaScript SDK

You can enable standard simulcast by setting simulcast: true in ConnectOptions when connecting to a video Room.

// Web JavaScript
// Remember that simulcast only needs to be enabled in media publishers
// See compatibility table above with supported browsers and required SDK versions

const room = await connect(token, {
    preferredVideoCodecs: [
      { codec: 'VP8', simulcast: true }
    ]
});

Any Group Room Participant with VP8 simulcast enabled publishes all their video tracks using VP8 simulcast. Once this is done, Twilio’s video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.

Enable simulcast using the iOS SDK

By default, simulcast is disabled. You can enable simulcast on a per-Participant basis when connecting to a Room. This is done using the ConnectOptions as shown in the following code snippet:

// Swift code
// Remember that simulcast only need to be enabled in media publishers
// See compatibility table above to with required SDK versions

let connectOptions = ConnectOptions(token: accessToken) { (builder) in
    builder.preferredVideoCodecs = [Vp8Codec(simulcast: true)]
}

Any Group Room Participant with VP8 simulcast enabled publishes all its video tracks using VP8 simulcast. Once this is done, Twilio’s video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.

Enable simulcast using the Android SDK

By default, simulcast is disabled. You can enable simulcast on a per-Participant basis when connecting to a Room. This is done using the ConnectOptions as shown in the following code snippet:

// Java code
// Remember that simulcast only need to be enabled in media publishers
// See compatibility table above to with required SDK versions

ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken).preferVideoCodecs(Collections.singletonList(new Vp8Codec(true))).build();

Any Group Room Participant with VP8 simulcast enabled publishes all its video tracks using VP8 simulcast. Once this is done, Twilio’s video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.

Simulcast and capture settings on mobile SDKs

To optimize video quality while minimizing CPU usage and bandwidth, it is recommended to use VP8 simulcast with the capture settings suggested below on each mobile platform.

iOS

Capture Frame Rate

24 FPS. When simulcasting, this will result in 3 temporal layers of 24 FPS, 12 FPS, and 6 FPS. Selecting 24 frames / second instead of the default of 30 reduces the CPU load on the VP8 software encoder.

Capture Dimensions

1. 1024x768 on most iPhones
2. 1280x720 on iPhone X and models that do not have support for 1024x768
3. 640x480 on iPhone 6s and earlier models

iOS devices support high resolution capture formats with ratios of 1.33:1 and 1.77:1. When simulcasting, it is often desirable to produce a squarish ratio (1.25:1) that can be viewed by subscribers in landscape or portrait, and as smaller thumbnails. Cropping is performed at the source by using a format request. Besides changing the ratio of the captured video, cropping also reduces the number of pixels that need to be processed by the software encoder. Using 1280x720 or 1024x768 for video capture will result in 3-layer simulcast with the layer structure as shown in the table above. Using 640x480 is recommended on older iPhones and will result in 2-layer simulcast.

Other Considerations

If a Group Room is being used, it is recommended to remove the rotation tags using hardware acceleration using this API. Also, it is recommended to reduce the audio bitrate tuned for speech content.

Sample Code

The above recommendations are implemented in this code snippet:

struct CaptureDeviceUtils {

    // Produce 3 spatial layers ~ {960x768, 480x384, 240x192}. 1024x768 is captured on most phones
    // Produce 3 spatial layers ~ {900x720, 450x360, 225x180}, 1280x720 is captured on on iPhone X
    static let kSimulcastVideoDimensions = CMVideoDimensions(width: 900, height: 720)
    static let kSimulcastVideoFrameRate = UInt(24)
    static let kSimulcastVideoBitrate = UInt(1800)

     /*
     * @brief Finds the smallest format that is suitably close to the ratio requested.
     *
     * @param device The AVCaptureDevice to query.
     * @param targetRatio The ratio that is preferred.
     *
     * @return A format that satisfies the request.
     */
    static func selectFormatBySize(device: AVCaptureDevice,
                                   targetSize: CMVideoDimensions) -> VideoFormat {
        // Arranged from smallest to largest.
        let formats = CameraSource.supportedFormats(captureDevice: device)
        var selectedFormat = formats.firstObject as? VideoFormat
        for format in formats {
            guard let videoFormat = format as? VideoFormat else {
                continue
            }
            if videoFormat.pixelFormat != PixelFormat.formatYUV420BiPlanarFullRange {
                continue
            }
            let dimensions = videoFormat.dimensions
            // Cropping might be used if there is not an exact match.
            if (dimensions.width >= targetSize.width && dimensions.height >= targetSize.height) {
                selectedFormat = videoFormat
                break
            }
        }
        return selectedFormat!
}

let options = CameraSourceOptions { (builder) in
    // Stripping rotation tags using hardware acceleration
    builder.rotationTags = .remove
}
camera = CameraSource(options: options, delegate: self)

// Assume front camera is available
let frontCamera = CameraSource.captureDevice(position: .front)
if let camera = camera {
    localVideoTrack = LocalVideoTrack(source: camera, enabled: true, name: "Camera")

    // Discover a simulcast format for the front camera
    let format = CaptureDeviceUtils.selectFormatBySize(device: frontCamera!,
                                                       targetSize: CaptureDeviceUtils.kSimulcastVideoDimensions)

    // Lower the frame rate to reduce CPU load, but still produce 3 temporal layers (f, f/2, f/4)
    format.frameRate = CaptureDeviceUtils.kSimulcastVideoFrameRate

    // Apply slight cropping to reduce CPU load, and provide square-ish video
    let croppedFormat = VideoFormat.init()
    croppedFormat.dimensions = CaptureDeviceUtils.kSimulcastVideoDimensions
    camera.requestOutputFormat(croppedFormat)

    camera.startCapture(device: device, format:format) { (captureDevice, videoFormat, error) in
        if let error = error {
            self.logMessage(messageText: "Capture failed with error.\ncode = \((error as NSError).code) error = \(error.localizedDescription)")
        }
    }
}

let connectOptions = ConnectOptions(token: accessToken) { (builder) in
    if let localVideoTrack = localVideoTrack {
        builder.videoTracks = [localVideoTrack]
    }
    builder.isNetworkQualityEnabled = true
    builder.networkQualityConfiguration =
        NetworkQualityConfiguration(localVerbosity: .minimal, remoteVerbosity: .minimal)
    // Enable Vp8 simulcast, and cap the bitrate at 1.8 Mbps to reduce strain on the sender. Reduce audio bitrate for speech content.
    builder.encodingParameters = EncodingParameters(audioBitrate:16, videoBitrate:1800)
    builder.preferredVideoCodecs = [Vp8Codec(simulcast: true)]
}

Android

Capture Frame Rate

24 FPS. When simulcasting, this will result in 3 temporal layers of 24 FPS, 12 FPS, and 6 FPS. Selecting 24 frames / second instead of the default of 30 reduces the CPU load on the VP8 encoder.

Capture Dimensions

1. 1280x720 on Android devices that support VP8 hardware acceleration
2. 1024x768 on more recent Android devices that do not support VP8 hardware acceleration
3. 640x480 on older Android devices

Using 1280x720 or 1024x768 for video capture will result in 3-layer simulcast with the layer structure as shown in the table above. Using 640x480 for video capture will result in a 2-layer simulcast.

Other Considerations

It is recommended to reduce the audio bitrate tuned for speech content.

Sample Code

The above settings are specified as part of the Video Constraints API as shown in the code snippet below:

import tvi.webrtc.MediaCodecVideoEncoder;

VideoDimensions videoDimensions = VideoDimensions.VGA_VIDEO_DIMENSIONS;
if (MediaCodecVideoEncoder.isVp8HwSupported()) {
    videoDimensions = VideoDimensions.HD_720P_VIDEO_DIMENSIONS;
}
VideoConstraints videoConstraints = new VideoConstraints.Builder()
                                                        .maxFps(VideoConstraints.FPS_24)
                                                        .maxVideoDimensions(videoDimensions)
                                                        .build();

LocalVideoTrack localVideoTrack = LocalVideoTrack.create(context, true, videoCapturer, videoConstraints);

// Enable network quality information for local and remote participants
NetworkQualityConfiguration configuration =
            new NetworkQualityConfiguration(
                        NetworkQualityVerbosity.NETWORK_QUALITY_VERBOSITY_MINIMAL,
                        NetworkQualityVerbosity.NETWORK_QUALITY_VERBOSITY_MINIMAL);

ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken)
            .enableNetworkQuality(true)
            .networkQualityConfiguration(configuration)
            .videoTracks(Collections.singletonList(localVideoTrack))
             // Cap the bitrate at 1.8 Mbps to reduce strain on the sender. Reduce audio bitrate for speech content.
            .encodingParameters(new EncodingParameters(16, 1800)
             // Enable Vp8 simulcast
            .preferVideoCodecs(Collections.singletonList(new Vp8Codec(true))) // Enable simulcast
            .build();