Menu

Expand
Rate this page:

Thanks for rating this page!

We are always striving to improve our documentation quality, and your feedback is valuable to us. How could this documentation serve you better?

Video Source APIs

In this guide, we will show you how to use the VideoSource APIs to share video in a Room. These APIs allow you to choose the built in camera(s), or any other source of content that is available to your application (or extension).

Overview

The VideoSource APIs describe producers and consumers of video content. A TVIVideoSource produces content for a TVILocalVideoTrack. Sources have the following properties.

  • VideoSources produce VideoFrames, and deliver them to VideoSinks.
  • VideoSources receieve format requests, and deliver requests to VideoSinks.
  • The recommended maximum frame size is 1920x1080.
  • The recommended maximum frame rate is 30 frames per second.
  • The recommended pixel format is NV12.

A TVIVideoSink consumes content from a TVIVideoSource. Sinks have the following properties.

  • VideoSinks handle format requests from VideoSources.
  • VideoSinks consume frames from VideoSources.

In the next section we will show you how to use the CameraSource API.

Using the CameraSource API

A TVICameraSource is a TVIVideoSource that produces content from the built-in cameras. This is probably the first kind of video that you want to share, so it is a good place to begin.

Create a CameraSource and a LocalVideoTrack

First we want to create a TVICameraSource, and use that source to create a TVILocalVideoTrack.

var cameraSource = TVICameraSource()
var videoTrack = TVILocalVideoTrack(source: cameraSource)

Capture from a Device

Now that we've setup our Track and Source, its time to start producing frames from one of the built-in cameras. Lets use a TVICameraSource utility method to help us discover a front facing AVCaptureDevice.

guard let frontCamera = TVICameraSource.captureDevice(for: .front) else {
    // The device does not have a front camera.
    return
}

// Start capturing with the device that we discovered.
cameraSource.startCapture(device: frontCamera)

In this example, TVICameraSource is automatically determining the best format to capture in. Typically, 640x480 at 30 frames / second is used as the default value.

Connect to a Room with a LocalVideoTrack

Next, we want to connect to a Room with the TVILocalVideoTrack we created earlier.

let connectOptions = TVIConnectOptions.init(token: accessToken){ (builder) in
    builder.roomName = "my-room"
    if let localVideoTrack = self.localVideoTrack {
        builder.videoTracks = [localVideoTrack]
    }
}
self.room = TwilioVideo.connect(with: connectOptions, delegate: self)

Select a new Device

While you can select a single device at start time, TVICameraSource also supports switching devices while it is already running. For example, you could switch from a front facing device to a rear facing device.

guard let rearCamera = TVICameraSource.captureDevice(for: .rear) else {
    // The device does not have a rear camera.
    return
}

cameraSource.select(device: rearCamera)

Unpublishing Video and Stopping Capture

At some point after connecting to a Room, you might decide that you want to stop sharing video from the camera. Start with unpublishing the Track.

// Unpublish the Track. We will no longer be sharing video in the Room.
if let participant = self.room?.localParticipant,
    let videoTrack = self.localVideoTrack {
    participant.unpublishVideoTrack(videoTrack)
}

Finally, we will stop the source and destroy the objects.

// Stop capturing from the device.
self.camera?.stopCapture(completion: { (error) in
    if let theError = error {
        print("Error stopping capture:", theError as Any)
    }

    self.camera = nil
    self.localVideoTrack = nil
})

Selecting a Device Format

An AVCaptureDevice can produce video in many possible formats. TVICameraSource offers utilty methods to discover formats that are suitable for video streaming. Consider executing the following code on your iOS device:

// Assume that we discovered "frontDevice" earlier.

let formats = TVICameraSource.supportedFormats(for: frontDevice)
print(formats)

When this code is run on an iPhone X with iOS 12.4, the following formats are returned.

Dimensions Frame Rate Pixel Format
192 x 144 30 420f
352 x 288 30 420f
480 x 360 30 420f
640 x 480 30 420f
960 x 540 30 420f
1280 x 720 30 420f
1920 x 1080 30 420f
1920 x 1440 30 420f
3088 x 2320 30 420f

Once you've determined which format you would like to use, you can provide it when starting capture.

// Formats are ordered by increasing dimensions. Pick the smallest size.
camera.startCapture(with: frontDevice, 
                  format: formats.firstObject as! TVIVideoFormat, 
              completion: nil)

In some applications, it may be important to change formats at runtime with as little disruption to the camera feed as possible.

// Select another format for the front facing camera.
camera.select(frontDevice,
              format: formats.lastObject as! TVIVideoFormat,
              completion: nil) 

Making a Format Request

Device formats afford quite a lot of flexibility, but there are some cases that AVCaptureDevice does not support out of the box. For example, what if you wanted to:

  1. Produce square video.
  2. Produce video that fills a portrait iPhone X / XR / XS screen.

These are both cases where you want to publish video in a different aspect ratio or size than AVCaptureDevice can produce. That is okay, because format requests are here to help with this problem.

let frontDevice = TVICameraSource.captureDevice(for: .front)!
let formats = TVICameraSource.supportedFormats(for: frontDevice)

// We match 640x480 directly, since it is known to be supported by all devices.
var preferredFormat: TVIVideoFormat?
for format in formats {
    let theFormat = format as! TVIVideoFormat
    if theFormat.dimensions.width == 640,
        theFormat.dimensions.height == 480 {
        preferredFormat = theFormat
    }
}

guard let captureFormat = preferredFormat else {
    // The perferred format could not be found.
    return
}

// Request cropping to 480x480.
let croppingRequest = TVIVideoFormat()
let dimension = captureFormat.dimensions.height
croppingRequest.dimensions = CMVideoDimensions(width: dimension,
                                               height: dimension)

self.camera?.requestOutputFormat(croppingRequest)
self.camera?.startCapture(with: frontDevice, 
                          format: captureFormat, 
                          completion: nil)

The following diagram shows the effect of a format request on frames produced by TVICameraSource.

Take a look at the iOS QuickStart Example to learn more about using TVICameraSource.

Writing a VideoSource

VideoSources are real-time producers of content. Importantly, to optimize for low latency delivery of individual frames is not guaranteed. The video pipeline continuously monitors network and device conditions and may respond by:

  • Reducing the number of bits allocated to the encoder.
  • Downscaling the video to a smaller size.
  • Dropping video frames at input.
  • Cropping (minimal, to ensure pixel alignment).

One benefit of a VideoSource over a VideoCapturer, is that format requests are considered at the same time as the video pipeline's requirements. This means that:

  • Cropping, and scaling operations are only performed once.
  • If the video pipeline drops a frame, the pixels need not be touched at all.

Sample Code

If you would like to explore writing your own VideoSource, then an excellent place to begin is with our sample code.

Rate this page:

Need some help?

We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd browsing the Twilio tag on Stack Overflow.