6.0 Changelog: Twilio Video Android

6.4.1 (September 13th, 2021)

• Programmable Video Android SDK 6.4.1
  • Maven Central
  • Documentation
• Programmable Video Android SDK KTX 6.4.1
  • Maven Central
  • Documentation

Bug Fixes

• Fixed a crash when maxTracks or renderDimensions is set and a sink is added for a RemoteVideoTrack.

Known Issues

• As of 6.0.0 of the SDK, hardware video encoding doesn't publish all of the three simulcast layers when VP8 simulcast is enabled on Android. This affects Video Content Preferences from working properly for subscribing video participants since the feature requires all three simulcast layers to switch between. Our team is working on a fix for this, but the feature does work when VP8 simulcast is used to publish video from participants using the Javascript or iOS SDKs.
• Video Content Preferences might prefer larger video than needed when device orientations are mismatched. For example, if a participant in landscape mode publishes video, then the subscribing participant must also be in landscape mode in order for the correctly sized simulcast layers to be selected. The same is true for the portrait orientation.
• When the publisher is publishing video at 720p with VP8 simulcast enabled and the subscriber varies their hints between 180p, 360p, and 720p, sometimes the subscriber receives larger video than expected.

Size Report

6.4.0 (August 3rd, 2021)

• Programmable Video Android SDK 6.4.0
  • Maven Central
  • Documentation
• Programmable Video Android SDK KTX 6.4.0
  • Maven Central
  • Documentation

New features

This release contains a significant update to the Bandwidth Profile API. It allows for more efficient use of bandwidth and CPU in multi-party applications. In addition, it provides developers with more dynamic control over which video tracks are delivered to the client and the preferred video resolution of the tracks. These capabilities are provided via the Client Track Switch Off Control and Video Content Preferences settings.

Existing Bandwidth Profile settings will continue to function as before, however, we recommend developers update their Bandwidth Profile settings to make use of these new capabilities at their earliest convenience.

Client Track Switch Off Control

• This feature allows subscribers to control whether the media for a RemoteVideoTrack is received or not. Client Track Switch off Control has two modes of operation:
• auto (default): The SDK determines whether tracks should be switched off based on renderer attachment, view visibility, and application lifecycle.
• manual: The application requests that individual tracks be switched off or on using the RemoteVideoTrack.switchOff() and RemoteVideoTrack.switchOn() methods.

Note If your application previously set the maxTracks property to limit the number of tracks visible, you should instead use clientTrackSwitchOffControl to take advantage of this feature.

Video Content Preferences

• This feature allows subscribers to specify preferences about the media that they receive on a RemoteVideoTrack. Video content preferences has two modes of operation:
• auto (default): The SDK specifies content preferences based on video view size. A RemoteVideoTrack rendered by a VideoView or VideoTextureView with larger dimensions will get a higher quality video compared to a RemoteVideoTrack rendered by a VideoView or VideoTextureView with smaller dimensions.
• manual: The application specifies the content preferences for individual tracks using RemoteVideoTrack.setContentPreferences().

Note If your application previously set the renderDimensions property, you should instead use contentPreferencesMode to take advantage of this feature.

Both of these features are available in Group Rooms and are enabled by default if your application specifies BandwidthProfileOptions during connect. The auto mode will be enabled if your application passes either a VideoTextureView or a VideoView to RemoteVideoTrack.addSink(VideoSink). However, if your application implements a custom VideoSink and it is passed to RemoteVideoTrack.addSink(VideoSink), then the track will remain switched on and video preferences are ignored as long as there is a custom VideoSink remaining in the track.

BandwidthProfileOptions bandwidthProfileOptions = new BandwidthProfileOptions(
  new VideoBandwidthProfileOptions.
    Builder()
    // Use "auto" default. Be sure to remove "maxTracks" and "renderDimensions".
    .build());
ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken)
  .bandwidthProfile(bandwidthProfileOptions)
  .build();

Video.connect(context, connectOptions, roomListener);

If you don't want the SDK to automatically switch on/off RemoteVideoTracks then specify ClientTrackSwitchOffControl.manual and VideoContentPreferencesMode.manual instead:

BandwidthProfileOptions bandwidthProfileOptions = new BandwidthProfileOptions(
  new VideoBandwidthProfileOptions.
    Builder()
      .clientTrackSwitchOffControl(ClientTrackSwitchOffControl.MANUAL)
      .videoContentPreferencesMode(VideoContentPreferencesMode.MANUAL)
      .build());
ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken)
  .bandwidthProfile(bandwidthProfileOptions)
  .build();

Video.connect(context, connectOptions, roomListener);

Subscribers that connect with ClientTrackSwitchOffControl.MANUAL can request which RemoteVideoTracks are switched on or off:

@Override
public void onVideoTrackSubscribed(
  RemoteParticipant remoteParticipant,
  RemoteVideoTrackPublication remoteVideoTrackPublication,
  RemoteVideoTrack remoteVideoTrack) {
    if (remoteParticipant.getIdentity() != "Bob") {
      remoteVideoTrack.switchOff();
    }
}

Subscribers that connect with VideoContentPreferencesMode.MANUAL can request which resolution they prefer to receive RemoteVideoTracks in:

@Override
public void onVideoTrackSubscribed(
  RemoteParticipant remoteParticipant,
  RemoteVideoTrackPublication remoteVideoTrackPublication,
  RemoteVideoTrack remoteVideoTrack) {
    remoteVideoTrack.setContentPreferences(new VideoContentPreferences(new VideoDimensions(320, 240)));
}

API changes

• The VideoBandwidthProfileOptions.maxTracks property is now deprecated and will raise a warning when set. Calling RemoteVideoTrack.switchOn() or RemoteVideoTrack.switchOff() after setting maxTracks is not allowed and will raise an exception.
• The VideoBandwidthProfileOptions.renderDimensions property is now deprecated and will raise a warning when set. Calling RemoteVideoTrack.setContentPreferences() after setting renderDimensions is not allowed and will raise an exception.

Known issues

• As of 6.0.0 of the SDK, hardware video encoding doesn't publish all of the three simulcast layers when VP8 simulcast is enabled on Android. This affects Video Content Preferences from working properly for subscribing video participants since the feature requires all three simulcast layers to switch between. Our team is working on a fix for this, but the feature does work when VP8 simulcast is used to publish video from participants using the JavaScript or iOS SDKs.
• Video Content Preferences might prefer larger video than needed when device orientations are mismatched. For example, if a participant in landscape mode publishes video, then the subscribing participant must also be in landscape mode in order for the correctly sized simulcast layers to be selected. The same is true for the portrait orientation.
• When the publisher is publishing video at 720p with VP8 simulcast enabled and the subscriber varies their hints between 180p, 360p, and 720p, sometimes the subscriber receives larger video than expected.

Size Report

6.3.1 (June 24th, 2021)

• Programmable Video Android SDK 6.3.1
  • Maven Central
  • Documentation
• Programmable Video Android SDK KTX 6.3.1
  • Maven Central
  • Documentation

Bug fixes

• Fixed a crash (see stack trace below) that sometimes occurs on Pixel devices when decoding VP8 video streams.

  java java.lang.IllegalArgumentException: Texture width must be positive, but was 0 FATAL EXCEPTION: AndroidVideoDecoder.outputThread Process: com.twilio.video.test, PID: 13001 java.lang.IllegalArgumentException: Texture width must be positive, but was 0 at tvi.webrtc.SurfaceTextureHelper.setTextureSize(SurfaceTextureHelper.java:256) at tvi.webrtc.AndroidVideoDecoder.deliverTextureFrame(AndroidVideoDecoder.java:432) at tvi.webrtc.AndroidVideoDecoder.deliverDecodedFrame(AndroidVideoDecoder.java:407) at tvi.webrtc.AndroidVideoDecoder$1.run(AndroidVideoDecoder.java:369)

• Fixed a bug that could cause a crash when changing log level to Trace at runtime.

Size report

6.3.0 (April 7th, 2021)

• Programmable Video Android SDK 6.3.0
  • Maven Central
  • Documentation
• Programmable Video Android SDK KTX 6.3.0
  • Maven Central
  • Documentation

Enhancements

• 100 Participant Group Rooms Pilot Program: A Group Room created with max participants greater than 50 is structured to support a small number of presenters and a large number of viewers. It has the following behavioral differences compared to regular Group Rooms:
  • The RoomListener.onParticipantConnected() callback method is invoked when a RemoteParticipant connects to the Room and publishes at least one Track.
  • The RoomListener.onParticipantDisconnected() callback method is invoked when a RemoteParticipant disconnects from the Room or unpublishes all of its Tracks.
  • If a RemoteParticipant unpublishes all of its tracks (resulting in the RoomListener.onParticipantDisconnected() callback method being invoked) and later republishes a track, a new RemoteParticipant object will be provided in the subsequent RoomListener.onParticipantConnected() callback method invocation with the same Participant Sid as before.
  • The maximum number of published Tracks in a Room at the same time cannot exceed 16. Attempts to publish more Tracks will result in a publication failure with TwilioException.PARTICIPANT_MAX_TRACKS_EXCEEDED_EXCEPTION unless one or more published Tracks is unpublished.
  • Contact your Twilio Account Executive to enroll in this pilot program.
• Updated ReLinker to 1.4.3 to consume from Maven Central.
• Now published to MavenCentral along with the Kotlin Extensions Library.

  • Ensure that you have mavenCentral listed in your project's buildscript repositories section:

    groovy buildscript { repositories { mavenCentral() ... } }

Size report

6.2.1 (January 25th, 2021)

• Programmable Video Android SDK 6.2.1
  • Bintray
  • Documentation
• Programmable Video Android SDK KTX 6.2.1
  • Bintray
  • Documentation

Bug fixes

• Fixed a crash that occurred while using either the CameraCapturer or Camera2Capturer, switching from the front camera to the back camera, and toggling the camera flash.

Known issues

• Publishing multiple tracks with the same name results in a crash if network quality is enabled. To avoid this, use unique names for each track in the Room.

Size Report

6.2.0 (January 13th, 2021)

• Programmable Video Android SDK 6.2.0
  • Bintray
  • Documentation
• Programmable Video Android SDK KTX 6.2.0
  • Bintray
  • Documentation

Enhancements

• Updated the SDK to Android Gradle Plugin 4.1.1

Maintenance

• Removed tvi.webrtc.NetworkMonitor and tvi.webrtc.NetworkMonitorAutoDetect from the libwebrtc.jar since they are unused by the SDK.

Size report

6.1.0 (December 17th, 2020)

• Programmable Video Android SDK 6.1.0
  • Bintray
  • Documentation
• Programmable Video Android SDK KTX 6.1.0
  • Bintray
  • Documentation

New Kotlin Extensions Library

As part of this release, we have created a new Kotlin Extensions Library. This library provides a set of convenience functions and extensions that provide Kotlin developers with a more idiomatic programming experience.

Extension APIs

Below is a list of all of the extension APIs that are now available:

• AudioOptions.kt
• BandwidthProfileOptions.kt
• ConnectOptions.kt
• DataTrackOptions.kt
• IceOptions.kt
• LocalAudioTrack.kt
• LocalDataTrack.kt
• LocalVideoTrack.kt
• Video.kt
• VideoBandwidthProfileOptions.kt

Getting started

To get started using the extensions, replace implementation com.twilio:video-android:6.1.0 with implementation com.twilio:video-android-ktx:6.1.0 in your application's build.gradle file. The extensions and functions are available under the com.twilio.video.ktx package. The extensions library also includes the SDK, so developers do not need to include the SDK and extensions library.

implementation 'com.twilio:video-android-ktx:6.1.0'

Versioning and support

The Video Android Kotlin extensions will follow the same semantic versioning policy of the Video Android SDK itself. Each Video Android release will now consist of the Java SDK and the Kotlin extensions library with the same version. Using the Kotlin extensions library is completely optional and developers can continue to use the Java based Video Android SDK as before.

All of the options extensions follow a similar pattern. For example, here's how to connect to a room with connect options using Java:

ConnectOptions connectOptions = new ConnectOptions.Builder(accessToken)
  .roomName("my-room")
  .enableAutomaticSubscription(false)
  .build();
room = Video.connect(context, connectOptions, roomListener);

The following Kotlin snippet shows how you can call connect with connect options all in one statement using the Video.kt connect extension function:

val room = Video.connect(context, token, roomListener) {
  roomName("my-room")
  enableAutomaticSubscription(false)
}

The library also introduces extension properties that make it easier to get and set specific Java fields. For example, here's how to check if a LocalVideoTrack track is enabled and how to disable it in Java:

LocalVideoTrack localVideoTrack = LocalVideoTrack.create(context, enable, cameraCapturer);

// Check if the track is enabled
localVideoTrack.isEnabled();

// Disable the track
localVideoTrack.enabled(false);

The following Kotlin snippet show how to do the same with the enabled extension property that is part of LocalVideoTrack.kt:

val localVideoTrack = LocalVideoTrack.create(context, enable, cameraCapturer)

// Check if the track is enabled
localVideoTrack.enabled

// Disable the track
localVideoTrack.enabled = false

Other enhancements

• The following classes now override the equals and hashCode object methods:
  • ConnectOptions.java
  • IceOptions.java
  • BandwidthProfileOptions.java
  • VideoBandwidthProfileOptions.java
  • AudioOptions.java
  • DataTrackOptions.java
• Added a new Video.java static method named getModuleLogLevel(LogModule module) that retrieves the log level for the specified LogModule.

Bug fixes

• Fixed a bug where publishing multiple tracks with the same name may result in a crash if network quality is enabled.

6.0.0 (December 4th, 2020)

SDK 6.0 is now generally available. 6.0 provides the following new features:

• The SDK has been upgraded to use Chromium WebRTC 83.
• The SDK now enables Discontinuous Transmission (DTX) by default when using the Opus codec. Enabling DTX results in a lower bitrate when silent audio is detected.
• The capturing and rendering facilities have been updated to allow developers more control, flexibility, and capabilities. This update includes the adoption of WebRTC classes.
• The SDK includes new callbacks onParticipantReconnecting and onParticipantReconnected to Room.Listener. These callbacks will be raised when a RemoteParticipant is attempting to reconnect to a room due to a signaling network interruption.

Please visit our migration guide for more details on how to update your application to 6.0.

Enhancements

• Added API documentation to Rgba8888Buffer.

Known issues

• Publishing multiple tracks with the same name results in a crash if network quality is enabled. To avoid this, use unique names for each track in the Room.

6.0.0-beta2 (December 2nd, 2020)

• Programmable Video Android SDK 6.0.0-beta2
  • Bintray
  • Documentation

Enhancements

• The SDK API docs now include documentation for classes defined in tvi.webrtc. For example, the VideoCapturer API docs will now provide a link to the base interface tvi.webrtc.VideoCapturer.
• isRecording() API now accurately reflects the current recording state of the Room. In the previous versions of the SDK, isRecording() could return false positives. The onRecordingStarted(...) and onRecordingStopped(...) callbacks will now be invoked when recording for at least a single track in the Room has started and stopped respectively.

Size Report

6.0.0-beta1 (November 23rd, 2020)

• Programmable Video Android SDK 6.0.0-beta1
  • Bintray
  • Documentation

Enhancements

• Added the ParticipantState enumeration.
• Added the getState() method to Participant.

• Added new callbacks onParticipantReconnecting and onParticipantReconnected to Room.Listener. These callbacks will be raised when a RemoteParticipant is attempting to reconnect to a room due to a signaling network interruption

  Note It can take up to 15 seconds for our signaling backend to detect that a RemoteParticipant's connection has been disrupted due to a network degradation or handoff.

API updates

• Removed the RemoteParticipant.isConnected() method in favor of Participant.getState().

Size Report

6.0.0-preview3 (November 17th, 2020)

• Programmable Video Android SDK 6.0.0-preview3
  • Bintray
  • Documentation

Enhancements

• Added useDtx parameter for constraining bitrate while using Opus codec. useDtx is enabled by default. Disabling useDtx will result in higher bitrate for silent audio while using Opus codec.

API updates

• OpusCodec class has a new constructor that takes useDtx as an input OpusCodec(boolean useDtx). Use this to explicitly enable or disable DTX.
• Removed IceOptions.abortOnIceServersTimeout and IceOptions.iceServersTimeout.
• Removed CameraCapturer.CameraSource. CameraCapturer instances are now created using a camera ID as a String. You can use tvi.webrtc.Camera1Enumerator#getDeviceNames() to query the list of supported camera IDs.

• Updated CameraCapturer#switchCamera() signature to require a camera ID. Users now must specify which camera ID to switch to. This behavior is similar to the Camera2Capturer switch camera behavior.

  The snippet below demonstrates the updated use of CameraCapturer.

  ```kotlin // Determine the front and back camera val camera1Enumerator = Camera1Enumerator() val frontCameraId = camera1Enumerator.deviceNames.find { camera1Enumerator.isFrontFacing(it) } val backCameraId = camera1Enumerator.deviceNames.find { camera1Enumerator.isBackFacing(it) }

  // Create the instance val cameraCapturer = CameraCapturer(context, frontCameraId)

  // Switch camera Ids cameraCapturer.switchCamera(backCameraId) ```

• Camera2Capturer.Listener implementations are now optional when creating a Camera2Capturer

6.0.0-preview2 (November 9th, 2020)

• Programmable Video Android SDK 6.0.0-preview2
  • Bintray
  • Documentation

AudioDevice API

Use the AudioDevice API to create innovative and advanced in-app audio capabilities. For example, enable pre-recorded messages to be played in-room, or, apply noise reduction algorithms before playing out the received audio.

An AudioDevice is a logical device that is used to capture and render (play out) audio. The captured audio is sent to the remote party, and the received audio is rendered locally. The current DefaultAudioDevice uses the mic for capturing and the local speaker/earpiece/headset for rendering. The AudioDevice API allows for the creation of custom audio capturers and renderers.

See this example and API docs to learn more about custom audio devices.

Enhancements

• Introduced the Video.audioDevice class member. You can now set your own AudioDevice before connecting to a Room.
• Added DefaultAudioDevice to render and capture audio. By default, the Video SDK uses DefaultAudioDevice to capture and render audio.
• Added the ability to provide custom audio capturing and rendering via the AudioDevice interface. AudioFormat describes the audio that is being captured and rendered.

6.0.0-preview1 (October 27th, 2020)

• The Video Android SDK uses Chromium WebRTC 83.
• The Video Android SDK is built with NDK r20b.

Adopting WebRTC Classes

The update to video capturing and rendering includes an adoption of WebRTC classes defined in libwebrtc.jar included with the Video Android SDK. In previous major releases, the SDK would define public APIs with classes defined exclusively in the com.twilio.video package. The SDK will now include public APIs with classes defined in libwebrtc.jar.

Any public APIs that reference a WebRTC class are only compatible with the libwebrtc.jar provided from the Video Android SDK. The classes provided in this artifact are defined under the tvi.webrtc package to ensure side-by-side compatibility with other WebRTC based libraries.

Adopting WebRTC classses in public APIs provides developers already familiar with WebRTC more flexibility when capturing media and rendering media.

Video Capturing Update

This release includes an overhaul to the video capturer facilities. LocalVideoTracks are now created with implementations of tvi.webrtc.VideoCapturer and VideoFormats. VideoCapturer remains a part of the SDK as an extension of tvi.webrtc.VideoCapturer, but VideoConstraints have been completely removed in favor of VideoFormats.

What's new

• Leverage capturers provided by libwebrtc.jar.
• Implement custom capturers that capture to textures.
• Added support for NV12 format.
• The video capturer API now enables cropping and scaling in response to network conditions.

Updated Programming Model

When a caller create a LocalVideoTrack, the following methods of a tvi.webrtc.VideoCapturer are called in the order listed.

1. isScreencast() — The return value is used to create a tvi.webrtc.VideoSource
2. initialize(surfaceTextureHelper, context, capturerObserver) - This method is called so the capturer can setup to capture frames. The capturer should retain a reference to the capturer observer.
3. startCapture(width, height, framerate) — The capturer should start capturing in a format as close as possible to width x height at framerate.

When a caller releases a local video track, the following methods are then called:

1. stopCapture() — The capturer should stop capturing frames. The SDK expects this method to block until frames have stopped being captured.
2. dispose() — The capturer should perform any final clean up.

API updates

• VideoPixelFormat has been removed in favor of implementing tvi.webrtc.VideoFrame.Buffer. The SDK includes support for the following buffer types: tvi.webrtc.VideoFrame.TextureBuffer, tvi.webrtc.NV21Buffer, tvi.webrtc.NV12Buffer, and Rgba8888Buffer.
• VideoCapturer now extends tvi.webrtc.VideoCapturer.
• getSupportedFormats() has been removed in favor of getCaptureFormat(). This update allows a VideoCapturer to provide a default capture format. For example, the ScreenCapturer returns a capture format based on the device screen dimensions.
• VideoConstraints has been removed. Developers can now specify a capture format when creating a local video track and the tvi.webrtc.VideoCapturer is responsible for capturing at a format as close as possible to the specified format. The VideoFormat provided to LocalVideoTrack.create(...) takes highest priority over the format returned by VideoCapturer#getCaptureFormat(). The default format for video tracks remains 640x480 at 30 FPS.
• Removed VideoFrame in favor of tvi.webrtc.VideoFrame.
• Removed AspectRatio.

Video Rendering Update

This release includes updated video rendering facilities. RemoteVideoTracks and LocalVideoTracks are now rendered with a tvi.webrtc.VideoSink instead of a VideoRenderer. VideoRenderer and VideoRenderer.Listener have been removed in favor of tvi.webrtc.VideoSink and tvi.webrtc.RendererCommon.RendererEvents respectively.

API updates

• Removed VideoRenderer in favor of tvi.webrtc.VideoSink.
• Removed VideoRenderer.Listener in favor of tvi.webrtc.RendererCommon.RendererEvents.
• Removed VideoTrack#addRenderer in favor of VideoTrack#addSink.
• Removed VideoTrack#removeRenderer in favor of VideoTrack#removeSink.
• Updated VideoView#setListener to take a tvi.webrtc.RendererCommon.RendererEvents instead of a VideoRenderer.Listener.
• Updated VideoTextureView#setListener to take a tvi.webrtc.RendererCommon.RendererEvents instead of a VideoRenderer.Listener.
• Removed CameraCapturer#takePicture. The functionality this API provides can be recreated using a custom VideoSink. Reference the Custom Video Sink quickstart for more details.

SDK size update

The table below highlights the updated app size impact.

Known issues

• AudioDevice API is not available in this release.