Skip to contentSkip to navigationSkip to topbar
Rate this page:
On this page

Video Source APIs


(warning)

Warning

This page is for reference only. We are no longer onboarding new customers to Programmable Video. Existing customers can continue to use the product until December 5, 2024(link takes you to an external page).
We recommend migrating your application to the API provided by our preferred video partner, Zoom. We've prepared this migration guide(link takes you to an external page) to assist you in minimizing any service disruption.

In this guide, we will show you how to use the VideoSource APIs to share video in a Room. These APIs allow you to choose the built in camera(s), or any other source of content that is available to your application (or extension).


Overview

overview page anchor

The VideoSource APIs describe producers and consumers of video content. A TVIVideoSource produces content for a TVILocalVideoTrack. Sources have the following properties.

  • VideoSources produce VideoFrames, and deliver them to VideoSinks.
  • VideoSources receive format requests, and deliver requests to VideoSinks.
  • The recommended maximum frame size is 1920x1080.
  • The recommended maximum frame rate is 30 frames per second.
  • The recommended pixel format is NV12.

A TVIVideoSink consumes content from a TVIVideoSource. Sinks have the following properties.

  • VideoSinks handle format requests from VideoSources.
  • VideoSinks consume frames from VideoSources.

In the next section we will show you how to use the CameraSource API.


Using the CameraSource API

using-the-camerasource-api page anchor

A TVICameraSource(link takes you to an external page) is a TVIVideoSource(link takes you to an external page) that produces content from the built-in cameras. This is probably the first kind of video that you want to share, so it is a good place to begin.

Create a CameraSource and a LocalVideoTrack

create-a-camerasource-and-a-localvideotrack page anchor

First we want to create a TVICameraSource, and use that source to create a TVILocalVideoTrack.


_10
guard let cameraSource = CameraSource() else {
_10
// Unable to initialize a camera source
_10
return
_10
}
_10
var videoTrack = LocalVideoTrack(source: cameraSource)

Now that we've setup our Track and Source, its time to start producing frames from one of the built-in cameras. Lets use a TVICameraSource utility method to help us discover a front facing AVCaptureDevice.


_10
guard let frontCamera = CameraSource.captureDevice(position: .front) else {
_10
// The device does not have a front camera.
_10
return
_10
}
_10
_10
// Start capturing with the device that we discovered.
_10
cameraSource.startCapture(device: frontCamera)

In this example, TVICameraSource is automatically determining the best format to capture in. Typically, 640x480 at 30 frames / second is used as the default value.

Connect to a Room with a LocalVideoTrack

connect-to-a-room-with-a-localvideotrack page anchor

Next, we want to connect to a Room with the TVILocalVideoTrack we created earlier.


_10
let connectOptions = ConnectOptions(token: accessToken){ (builder) in
_10
builder.roomName = "my-room"
_10
if let localVideoTrack = self.localVideoTrack {
_10
builder.videoTracks = [localVideoTrack]
_10
}
_10
}
_10
self.room = TwilioVideoSDK.connect(options: connectOptions, delegate: self)

While you can select a single device at start time, TVICameraSource also supports switching devices while it is already running. For example, you could switch from a front facing device to a rear facing device.


_10
guard let rearCamera = CameraSource.captureDevice(position: .back) else {
_10
// The device does not have a rear camera.
_10
return
_10
}
_10
_10
cameraSource.selectCaptureDevice(rearCamera)

Unpublishing Video and Stopping Capture

unpublishing-video-and-stopping-capture page anchor

At some point after connecting to a Room, you might decide that you want to stop sharing video from the camera. Start with unpublishing the Track.


_10
// Unpublish the Track. We will no longer be sharing video in the Room.
_10
if let participant = self.room?.localParticipant,
_10
let videoTrack = self.localVideoTrack {
_10
participant.unpublishVideoTrack(videoTrack)
_10
}

Finally, we will stop the source and destroy the objects.


_10
// Stop capturing from the device.
_10
self.camera?.stopCapture(completion: { (error) in
_10
if let theError = error {
_10
print("Error stopping capture:", theError as Any)
_10
}
_10
_10
self.camera = nil
_10
self.localVideoTrack = nil
_10
})

Selecting a Device Format

selecting-a-device-format page anchor

An AVCaptureDevice can produce video in many possible formats. TVICameraSource offers utility methods to discover formats that are suitable for video streaming. Consider executing the following code on your iOS device:


_10
// Assume that we discovered "frontDevice" earlier.
_10
_10
let formats = CameraSource.supportedFormats(captureDevice: frontDevice)
_10
print(formats)

When this code is run on an iPhone X with iOS 12.4, the following formats are returned.

DimensionsFrame RatePixel Format
192 x 14430420f
352 x 28830420f
480 x 36030420f
640 x 48030420f
960 x 54030420f
1280 x 72030420f
1920 x 108030420f
1920 x 144030420f
3088 x 232030420f

Once you've determined which format you would like to use, you can provide it when starting capture.


_10
// Formats are ordered by increasing dimensions. Start with the smallest size.
_10
cameraSource.startCapture(device: frontDevice,
_10
format: formats.firstObject as! VideoFormat,
_10
completion: nil)

In some applications, it may be important to change formats at runtime with as little disruption to the camera feed as possible.


_10
// Select another format for the front facing camera.
_10
cameraSource.selectCaptureDevice(frontDevice,
_10
format: formats.lastObject as! VideoFormat,
_10
completion: nil)

Device formats afford quite a lot of flexibility, but there are some cases that AVCaptureDevice does not support out of the box. For example, what if you wanted to:

  1. Produce square video.
  2. Produce video that fills a portrait iPhone X / XR / XS screen.

These are both cases where you want to publish video in a different aspect ratio or size than AVCaptureDevice can produce. That is okay, because format requests are here to help with this problem.


_28
let frontDevice = CameraSource.captureDevice(position: .front)!
_28
let formats = CameraSource.supportedFormats(captureDevice: frontDevice)
_28
_28
// We match 640x480 directly, since it is known to be supported by all devices.
_28
var preferredFormat: VideoFormat?
_28
for format in formats {
_28
let theFormat = format as! VideoFormat
_28
if theFormat.dimensions.width == 640,
_28
theFormat.dimensions.height == 480 {
_28
preferredFormat = theFormat
_28
}
_28
}
_28
_28
guard let captureFormat = preferredFormat else {
_28
// The preferred format could not be found.
_28
return
_28
}
_28
_28
// Request cropping to 480x480.
_28
let croppingRequest = VideoFormat()
_28
let dimension = captureFormat.dimensions.height
_28
croppingRequest.dimensions = CMVideoDimensions(width: dimension,
_28
height: dimension)
_28
_28
self.camera?.requestOutputFormat(croppingRequest)
_28
self.camera?.startCapture(device: frontDevice,
_28
format: captureFormat,
_28
completion: nil)

The following diagram shows the effect of a format request on frames produced by TVICameraSource.

Take a look at the iOS QuickStart Example(link takes you to an external page) to learn more about using TVICameraSource.


Tracking Orientation Changes

tracking-orientation-changes page anchor

The TVICameraSource provides flexibility in how it tracks video orientation for capture and preview. By default, the TVICameraSource monitors -[UIApplication statusBarOrientation] for orientation changes. With the addition of the UIWindowScene APIs in iOS 13, TVICameraSource now has a property, TVICameraSourceOptions.orientationTracker, which allows you to specify how the TVICameraSource should track orientation changes.

The orientationTracker property accepts an object that implements the TVICameraSourceOrientationTracker protocol. A default implementation, TVIUserInterfaceTracker is provided with the SDK. TVIUserInterfaceTracker monitors for changes in UIInterfaceOrientation at the application or scene level. For example, if you wish to track orientation changes based on a scene, you would provide the scene to track when creating the TVICameraSourceOptions.


_10
// Track the orientation of the key window's scene.
_10
let options = CameraSourceOptions { (builder) in
_10
if let keyScene = UIApplication.shared.keyWindow?.windowScene {
_10
builder.orientationTracker = UserInterfaceTracker(scene: keyScene)
_10
}
_10
}
_10
let camera = CameraSource(options: options, delegate: self)

You will also need to forward UIWindowScene events from your UIWindowSceneDelegate to keep TVIUserInterfaceTracker up to date as the scene changes.


_10
// Forward UIWindowScene events
_10
func windowScene(_ windowScene: UIWindowScene,
_10
didUpdate previousCoordinateSpace: UICoordinateSpace,
_10
interfaceOrientation previousInterfaceOrientation: UIInterfaceOrientation,
_10
traitCollection previousTraitCollection: UITraitCollection) {
_10
UserInterfaceTracker.sceneInterfaceOrientationDidChange(windowScene)
_10
}

You can also manually control how orientation is tracked. For example, you might decide to use UIDevice instead of UIScene to determine the orientation of the camera. To do this, you would create your own implementation of TVICameraSourceOrientationTracker which invokes the - (void)trackerOrientationDidChange:(AVCaptureVideoOrientation)orientation callback method when the device's orientation changes.


VideoSources are real-time producers of content. Importantly, to optimize for low latency delivery of individual frames is not guaranteed. The video pipeline continuously monitors network and device conditions and may respond by:

  • Reducing the number of bits allocated to the encoder.
  • Downscaling the video to a smaller size.
  • Dropping video frames at input.
  • Cropping (minimal, to ensure pixel alignment).

If you would like to implement your own VideoSource, or learn about advanced usage of TVICameraSource then an excellent place to begin is with our sample code.


Rate this page: