Working with VP8 Simulcast

Simulcast is a standardized technique used to retain media quality when some subscribers have limited bandwidth. It is a mechanism for providing scalability to non-scalable video codecs such as VP8.

With simulcast, a client sends multiple versions of the same video simultaneously. Each version is encoded independently at a different resolution and frame rate; this way, a subscriber with limited bandwidth can receive a lower quality version of the media, but subscribers with more bandwidth can receive a higher quality video and their media quality is not degraded.

Twilio Video provides the option to use VP8 simulcast via Twilio's Selective Forwarding Unit (SFU).

SFUs only forward media and can neither transcode nor modify the video. When sending media with unicast (only sending one version of the video track), publishers need to reduce quality to adapt to the worst of their subscribers' bandwidths so that no subscriber is congested.

With VP8 simulcast, the SFU can forward higher quality videos to higher bandwidth subscribers and lower quality videos to lower bandwidth ones. The track publisher sends different track qualities and the SFU selects the most optimal quality for each subscriber. Then, the subscriber receives a single VP8 encoded video that is most suited for their network conditions.

The following illustration shows the difference between unicast and simulcast.

Expand image

Simulcast offers the following benefits Participants:

• VP8 subscribers can receive video adapted to their available bandwidth. This significantly improves the quality in Rooms with many heterogeneous Participants.
• VP8 subscribers are isolated from each other so that a subscriber with a degraded network link does not affect the reception quality of other subscribers.

On the other hand, simulcast also has some drawbacks:

• Simulcast only contributes to improve the video quality in Rooms with three or more Participants.
• On mobile devices, publishers' battery consumption is higher because the publisher encodes multiple versions of the same video track.
• Publishers' bandwidth consumption is higher (up to double in some cases) because the publisher sends multiple versions of the same video track. This increase does not impact your Programmable Video costs as Twilio does not charge for upstream bandwidth (i.e. from the sender to Twilio's cloud).

Twilio's standard VP8 simulcast sends up to three layers of video at different resolutions. See the approximate resolutions of each layer here. In some video conferencing contexts, the higher resolution layers that consume the most resources to encode may not be needed.

Twilio Video offers adaptive simulcast, which enables and disables simulcast layers dynamically to improve bandwidth and CPU usage. This helps save device resources in cases such as presentation and grid modes, when the application does not need a Participant's highest resolution video. Adaptive simulcast ensures that publishers are only encoding the spatial layers needed at a given moment.

For example, when someone is presenting in a video conference, you will frequently only display the presenter's video in a large format, and will display only thumbnails of other participants' video. The participants who are not presenting do not need to encode and send higher resolution video layers because their video is not highlighted. The same might also be true in a video conference in grid mode, where each participant's video is the same size and no one's video needs to be the highest quality.

In these situations, adaptive simulcast can detect which layers are being used by subscribers and automatically turn off encoding on the publisher's side for higher spatial layers that are not being used. As speakers change, adaptive simulcast will dynamically turn on or turn off the appropriate spatial layers, based on what subscribers in the room are using.

Note that adaptive simulcast will not disable any video layers when the Room is being recorded, to help produce high quality recordings.

Adaptive simulcast is currently available in the Twilio Video JavaScript SDK, Android SDK, and iOS SDK. Please review:

• Enable adaptive simulcast using the JavaScript SDK
• Enable adaptive simulcast using the Android SDK
• Enable adaptive simulcast using the iOS SDK

for more information. If your application is currently using VP8 simulcast, we recommend that you switch to adaptive simulcast.

• Simulcast is only supported for the VP8 video codec.

Twilio SDKs encode up to three spatial layers when simulcast is enabled. The following table illustrates which layers are typically generated given a particular capture resolution. Remark that this is just an approximation and that the real behavior may be slightly different. In the table, disabled means that layer is not sent in those conditions. (Video of the specified resolution is not generated by the publisher and is not available at the SFU to be forwarded to subscribers).

The following table illustrates Twilio's current support for simulcast:

Simulcast is disabled by default. You can enable simulcast on a per-Participant basis when connecting to a Room.

To enable adaptive simulcast, set preferredVideoCodecs="auto" in ConnectOptions when connecting to a video Room. The SDK will use VP8 simulcast, and will enable/disable simulcast layers dynamically, thus improving bandwidth and CPU usage.

Adaptive simulcast works best when used along with Client Track Switch Off Control and Video Content Preferences. These two flags allow the SFU to determine which simulcast layers are needed, thus allowing it to disable the layers not needed on publisher side.

1
const { connect } = require("twilio-video");
2

3
const room = await connect(token, {
4
  preferredVideoCodecs: "auto",
5
  bandwidthProfile: {
6
    video: {
7
      contentPreferencesMode: "auto",
8
      clientTrackSwitchOffControl: "auto",
9
    },
10
  },
11
});

Note: following limitations with adaptive simulcast in the JavaScript SDK:

• Specifying preferredVideoCodecs="auto" will revert to unicast in the following cases:
  • The publisher is using Firefox
  • The publisher has preferred the H264 codec
  • The Room is configured to support only the H264 codec
• When the Room is being recorded, the SFU will not disable any simulcast layers of the publisher's VideoTrack.

You can enable standard simulcast by setting simulcast: true in ConnectOptions when connecting to a video Room.

1
// Web JavaScript
2
// Remember that simulcast only needs to be enabled in media publishers
3
// See compatibility table above with supported browsers and required SDK versions
4

5
const room = await connect(token, {
6
  preferredVideoCodecs: [{ codec: "VP8", simulcast: true }],
7
});

Any Participant with VP8 simulcast enabled publishes all their video tracks using VP8 simulcast. Once this is done, Twilio's video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.

By default, simulcast is disabled. You can enable simulcast on a per-Participant basis when connecting to a Room. This is done using the ConnectOptions as shown in the following code snippet:

1
// Swift code
2
// Remember that simulcast only need to be enabled in media publishers
3
// See compatibility table above to with required SDK versions
4

5
let connectOptions = ConnectOptions(token: accessToken) { (builder) in
6
    builder.preferredVideoCodecs = [Vp8Codec(simulcast: true)]
7
}

Any Participant with VP8 simulcast enabled publishes all its video tracks using VP8 simulcast. Once this is done, Twilio's video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.

By default, simulcast is disabled. You can enable simulcast on a per-Participant basis when connecting to a Room. This is done using the ConnectOptions as shown in the following code snippet:

1
// Java code
2
// Remember that simulcast only need to be enabled in media publishers
3
// See compatibility table above to with required SDK versions
4

5
ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken).preferVideoCodecs(Collections.singletonList(new Vp8Codec(true))).build();

Any Participant with VP8 simulcast enabled publishes all its video tracks using VP8 simulcast. Once this is done, Twilio's video infrastructure leverages simulcast tracks to provide the best possible quality to any subscriber without requiring any additional action from you.

To optimize video quality while minimizing CPU usage and bandwidth, it is recommended to use VP8 simulcast with the capture settings suggested below on each mobile platform.

24 FPS. When simulcasting, this will result in 3 temporal layers of 24 FPS, 12 FPS, and 6 FPS. Selecting 24 frames / second instead of the default of 30 reduces the CPU load on the VP8 software encoder.

1. 1024x768 on most iPhones
2. 1280x720 on iPhone X and models that do not have support for 1024x768
3. 640x480 on iPhone 6s and earlier models

iOS devices support high resolution capture formats with ratios of 1.33:1 and 1.77:1. When simulcasting, it is often desirable to produce a squarish ratio (1.25:1) that can be viewed by subscribers in landscape or portrait, and as smaller thumbnails. Cropping is performed at the source by using a format request. Besides changing the ratio of the captured video, cropping also reduces the number of pixels that need to be processed by the software encoder. Using 1280x720 or 1024x768 for video capture will result in 3-layer simulcast with the layer structure as shown in the table above. Using 640x480 is recommended on older iPhones and will result in 2-layer simulcast.

It is recommended to remove the rotation tags using hardware acceleration using this API. Also, it is recommended to reduce the audio bitrate tuned for speech content.

The above recommendations are implemented in this code snippet:

1
struct CaptureDeviceUtils {
2

3
    // Produce 3 spatial layers ~ {960x768, 480x384, 240x192}. 1024x768 is captured on most phones
4
    // Produce 3 spatial layers ~ {900x720, 450x360, 225x180}, 1280x720 is captured on iPhone X
5
    static let kSimulcastVideoDimensions = CMVideoDimensions(width: 900, height: 720)
6
    static let kSimulcastVideoFrameRate = UInt(24)
7
    static let kSimulcastVideoBitrate = UInt(1800)
8

9
     /*
10
     * @brief Finds the smallest format that is suitably close to the ratio requested.
11
     *
12
     * @param device The AVCaptureDevice to query.
13
     * @param targetRatio The ratio that is preferred.
14
     *
15
     * @return A format that satisfies the request.
16
     */
17
    static func selectFormatBySize(device: AVCaptureDevice,
18
                                   targetSize: CMVideoDimensions) -> VideoFormat {
19
        // Arranged from smallest to largest.
20
        let formats = CameraSource.supportedFormats(captureDevice: device)
21
        var selectedFormat = formats.firstObject as? VideoFormat
22
        for format in formats {
23
            guard let videoFormat = format as? VideoFormat else {
24
                continue
25
            }
26
            if videoFormat.pixelFormat != PixelFormat.formatYUV420BiPlanarFullRange {
27
                continue
28
            }
29
            let dimensions = videoFormat.dimensions
30
            // Cropping might be used if there is not an exact match.
31
            if (dimensions.width >= targetSize.width && dimensions.height >= targetSize.height) {
32
                selectedFormat = videoFormat
33
                break
34
            }
35
        }
36
        return selectedFormat!
37
}
38

39
let options = CameraSourceOptions { (builder) in
40
    // Stripping rotation tags using hardware acceleration
41
    builder.rotationTags = .remove
42
}
43
camera = CameraSource(options: options, delegate: self)
44

45
// Assume front camera is available
46
let frontCamera = CameraSource.captureDevice(position: .front)
47
if let camera = camera {
48
    localVideoTrack = LocalVideoTrack(source: camera, enabled: true, name: "Camera")
49

50
    // Discover a simulcast format for the front camera
51
    let format = CaptureDeviceUtils.selectFormatBySize(device: frontCamera!,
52
                                                       targetSize: CaptureDeviceUtils.kSimulcastVideoDimensions)
53

54
    // Lower the frame rate to reduce CPU load, but still produce 3 temporal layers (f, f/2, f/4)
55
    format.frameRate = CaptureDeviceUtils.kSimulcastVideoFrameRate
56

57
    // Apply slight cropping to reduce CPU load, and provide square-ish video
58
    let croppedFormat = VideoFormat.init()
59
    croppedFormat.dimensions = CaptureDeviceUtils.kSimulcastVideoDimensions
60
    camera.requestOutputFormat(croppedFormat)
61

62
    camera.startCapture(device: device, format:format) { (captureDevice, videoFormat, error) in
63
        if let error = error {
64
            self.logMessage(messageText: "Capture failed with error.\ncode = \((error as NSError).code) error = \(error.localizedDescription)")
65
        }
66
    }
67
}
68

69
let connectOptions = ConnectOptions(token: accessToken) { (builder) in
70
    if let localVideoTrack = localVideoTrack {
71
        builder.videoTracks = [localVideoTrack]
72
    }
73
    builder.isNetworkQualityEnabled = true
74
    builder.networkQualityConfiguration =
75
        NetworkQualityConfiguration(localVerbosity: .minimal, remoteVerbosity: .minimal)
76
    // Enable Vp8 simulcast, and cap the bitrate at 1.8 Mbps to reduce strain on the sender. Reduce audio bitrate for speech content.
77
    builder.encodingParameters = EncodingParameters(audioBitrate:16, videoBitrate:1800)
78
    builder.preferredVideoCodecs = [Vp8Codec(simulcast: true)]
79
}

24 FPS. When simulcasting, this will result in 3 temporal layers of 24 FPS, 12 FPS, and 6 FPS. Selecting 24 frames / second instead of the default of 30 reduces the CPU load on the VP8 encoder.

1. 1280x720 on Android devices that support VP8 hardware acceleration
2. 1024x768 on more recent Android devices that do not support VP8 hardware acceleration
3. 640x480 on older Android devices

Using 1280x720 or 1024x768 for video capture will result in 3-layer simulcast with the layer structure as shown in the table above. Using 640x480 for video capture will result in a 2-layer simulcast.

It is recommended to reduce the audio bitrate tuned for speech content.

The above settings are specified as part of the Video Format API as shown in the code snippet below:

1
import tvi.webrtc.MediaCodecVideoEncoder;
2

3
VideoDimensions videoDimensions = VideoDimensions.VGA_VIDEO_DIMENSIONS;
4
if (MediaCodecVideoEncoder.isVp8HwSupported()) {
5
    videoDimensions = VideoDimensions.HD_720P_VIDEO_DIMENSIONS;
6
}
7
VideoFormat videoFormat = new VideoFormat(videoDimensions, 24);
8

9
LocalVideoTrack localVideoTrack = LocalVideoTrack.create(context, true, videoCapturer, videoFormat);
10

11
// Enable network quality information for local and remote participants
12
NetworkQualityConfiguration configuration =
13
            new NetworkQualityConfiguration(
14
                        NetworkQualityVerbosity.NETWORK_QUALITY_VERBOSITY_MINIMAL,
15
                        NetworkQualityVerbosity.NETWORK_QUALITY_VERBOSITY_MINIMAL);
16

17
ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken)
18
            .enableNetworkQuality(true)
19
            .networkQualityConfiguration(configuration)
20
            .videoTracks(Collections.singletonList(localVideoTrack))
21
             // Cap the bitrate at 1.8 Mbps to reduce strain on the sender. Reduce audio bitrate for speech content.
22
            .encodingParameters(new EncodingParameters(16, 1800)
23
             // Enable Vp8 simulcast
24
            .preferVideoCodecs(Collections.singletonList(new Vp8Codec(true))) // Enable simulcast
25
            .build();