Skip to contentSkip to navigationSkip to topbar
Rate this page:
On this page

Understanding Video Rooms APIs


(warning)

Warning

This page is for reference only. We are no longer onboarding new customers to Programmable Video. Existing customers can continue to use the product until December 5, 2024(link takes you to an external page).
We recommend migrating your application to the API provided by our preferred video partner, Zoom. We've prepared this migration guide(link takes you to an external page) to assist you in minimizing any service disruption.

In the Basic Concepts Guide we explain that Video Rooms applications are split into:

  • Application server-side code: authenticates end-users and contains your business logic.
  • Application client-side code: captures, exchanges, and renders the media allowing end-users to communicate in real-time.

Developers can create applications using three Twilio APIs, as the following picture illustrates:

  1. The Rooms REST API (server-side)
  2. The Status Callbacks API (server-side)
  3. The Video Client SDK API (client-side)
Anatomy of a Twilio Video Application.

The Rooms REST API

the-rooms-rest-api page anchor

The Rooms REST API is based on the following REST resources and dependencies:

REST Room API UML Object Model.

Room

room page anchor

A Room represents a virtual space where end-users communicate. Technically, a Room is a computing resource that provides Real-time Communications (RTC) services to client applications through a set of APIs. More specifically, a Room provides:

  • A session: so that end-users can connect/disconnect to/from Rooms. When an end-user connects, we say it is a Room Participant.
  • An RTC Service: so that Participants can exchange audio, video, and data tracks using WebRTC.

In Twilio, a Room is identified by a SID: a string that uniquely identifies that Room inside Twilio. A Room SID always start with the characters RM and looks like this: RMdba2b037253f3a9a81a591681e88ce96.

Twilio Rooms REST API allows developers to:

You can review the Understanding Video Rooms Guide and the Rooms REST API Reference Documentation for more information about Rooms.

A Participant represents a client (i.e. an end-user) that is connected to a Room and can use the Room's communication capabilities. In a given Room there can be zero (i.e. an empty Room) or more Participants. A Participant can be connected only to one Room.

In Twilio, a Participant is uniquely identified by a SID with a PA prefix like the following: PA734b33d8bde2a3b3c7720964c87961c9.

Twilio Rooms REST API allows developers to:

A Track is a stream of bytes that contain the data generated by a multimedia source such as a microphone or a camera. Twilio Rooms are based on a publish/subscribe model. This means that a Participant can publish media tracks to the Room. The rest of the Room's Participants can then subscribe to these tracks and start receiving the media information.

In Twilio, a Track is uniquely identified by a SID with the MT prefix like the following: MT6e87a66bd033cdced4efe1267c595a16.

For a given Participant, the Twilio Rooms REST API allows developers to:

A SubscribeRule is a specification of the Tracks a given Participant should subscribe to. For a given Participant, the Twilio Rooms REST API allows developers to:

Using the Rooms REST API

using-the-rooms-rest-api page anchor

Developers can use the Rooms REST API directly firing the HTTP request to the appropriate endpoints. As an alternative, Twilio has created server-side helper libraries that expose all our REST APIs capabilities in a consistent and seamless way.


The Status Callbacks API

the-status-callbacks-api page anchor

Status Callbacks allow you to receive events related to the REST resources managed by Twilio. In order to use Status Callbacks you need to configure your Application Server to be able to receive HTTP requests issued by Twilio. Status callbacks events will notify your application when relevant events occur to your Rooms, Participants and Tracks indicating the appropriate SIDs to identify the affected resources. Check the Programmable Video Status Callbacks Guide for further information.


The Video Client SDK API

the-video-client-sdk-api page anchor

Video Rooms SDKs are available on three different platforms: JavaScript, Android, and iOS. These SDKs share similar type hierarchies based on the following classes: Room, Participant, TrackPublication, and Track. The following UML diagram(link takes you to an external page) shows their relationship.

Video Client SDK API UML Object Model - Room, Participants and Tracks.

This type represent Rooms from the perspective of the client. When a client connects to a Twilio Room it obtains a Room object instance. The Room class has primitives and events that developers can use to:

  • Get the Room name and SID
  • Get notified when participants connect or disconnect
  • Disconnect from the Room.

This snippet shows how to connect to a Room using the JavaScript SDK:


_10
const {connect} = require('twilio-video');
_10
connect('$TOKEN', { name: 'my-room' }).then(room => {
_10
console.log('I connected to Room ${room.name} that has SID ${room.sid}')
_10
}, error =>{
_10
console.error('Unable to connect to Room: ${error.message}')
_10
})

This type represents a Participant from the perspective of the client. A Room can have zero or more Participants.

A TrackPublication represents the communication of a media Track from a Participant to the Room. A Participant can have zero or more TrackPublications.

This type represents a Track from the perspective of the client.

Understanding Participant Types (Client SDK API)

understanding-participant-types-client-sdk-api page anchor

Our client SDK APIs are designed in "first person". This means that developers should think about them from the perspective of the client where the code is running. From that perspective, there are two types of Participants as illustrated in the following diagram:

Video SDK Participant Types - LocalParticipant and RemoteParticipant.

LocalParticipant

localparticipant page anchor

The LocalParticipant is the Participant on "this" client (i.e. on the client the code is running). The LocalParticipant can access the client hardware resources and manages the status and preferences of the client with respect to the Room. Developers can obtain a reference to the LocalParticipant from the Room instance, as the following code illustrates:


_10
localParticipant = room.localParticipant
_10
console.log('The LocalParticipant identity is ${localparticipant.identity}')
_10
console.log('The LocalParticipant SID is ${localParticipant.sid}')

RemoteParticipants represent all other Participants in the Room that are not on this specific client. RemoteParticipant objects can be seen as stubs that represent remote client status and capabilities on "this" client. RemoteParticipants can be obtained from the Room, as the following code illustrates:


_11
//List RemoteParticipants connected to the Room
_11
room.participants.forEach(remoteParticipant => {
_11
console.log('RemoteParticipant ${remoteParticipant.identity} is connected')
_11
console.log('RemoteParticipant SID is ${remoteParticipant.sid}')
_11
});
_11
_11
//Get notified when a RemoteParticipant connects to the Room
_11
room.on('participantConnected', remoteParticipant => {
_11
console.log('RemoteParticipant ${remoteParticipant.identity} just connected')
_11
console.log('RemoteParticipant SID is ${remoteParticipant.sid}')
_11
});

Understanding Track Types (Client SDK API)

understanding-track-types-client-sdk-api page anchor

Our SDKs support three Track subtypes:

  • AudioTrack: It represents a stream of bytes being generated by an audio source such as a microphone.
  • VideoTrack: It represents a stream of bytes being generated by a video source such as a webcam or a screen.
  • DataTrack: It represents a stream of bytes (or characters) that are generated by the application. In other words, developers can use a DataTrack to communicate arbitrary data among participants with very low latency. Notice that all Tracks being published by a Participant share the same underlying transport. Hence, the latency properties of the DataTrack bytes published by a Participant should be the the same than the latency properties of the audio and video information published by the Participant.
Programmable Video SDK Track Types.

Following the above mentioned "first person" API mode, each Track subtype has two versions:

LocalAudioTrack, LocalVideoTrack and LocalDataTrack are the local Track subtypes. Be aware that:

  • Instances of these types can only be created at the local SDK.
  • Local Tracks do not have a SID because they are not known by Twilio (i.e. they are local to the SDK they have been created in). Tracks are only assigned a SID when they are published.
  • Local Tracks have an ID that identifies them locally (i.e. the ID of a published Local Track does not match with the ID of the subscribed Remote Track).
  • Local Tracks can be assigned a name at creation time.
  • LocalVideoTrack can be attached to the local UI to create a camera preview.

_10
const { createLocalVideoTrack, createLocalAudioTrack } = require('twilio-video');
_10
createLocalAudioTrack({name:'john-audio-track').then(localAudioTrack => {
_10
console.log('Created LocalAudioTrack with id ${localAudioTrack.id}')
_10
})
_10
createLocalVideoTrack({name:'alices-webcam'}).then(localVideoTrack => {
_10
console.log('Created LocalVideoTrack with id ${localVideoTrack.id}')
_10
const localMediaContainer = document.getElementById('local-media');
_10
localMediaContainer.appendChild(localVideoTrack.attach());
_10
})

RemoteAudioTrack, RemoteVideoTrack and RemoteDataTrack are the remote Track subtypes.

  • Instances of these types can only exists at the RemoteParticipants that are subscribed to them.
  • A Local Track published at a given Participant will be seen as a Remote Track by the rest of subscribed Participants.
  • Remote Tracks always have a SID that matches with the SID assigned to the corresponding published Track
  • Remote Tracks have a name that matches with the name assigned to the corresponding published Track.
  • Remote Tracks are held by the owning RemoteParticipant

_10
remoteParticipant.on('trackSubscribed', remoteTrack => {
_10
console.log('Received a ${remoteTrack.kind} track with SID ${remoteTrack.sid}')
_10
document.getElementById('remote-media-div').appendChild(track.attach());
_10
});

Understanding TrackPublication Types (Client SDK API)

understanding-trackpublication-types-client-sdk-api page anchor

As with Track, TrackPublication also has three media-dependent subtypes:

  • AudioTrackPublication: it represents the publication of an AudioTrack.
  • VideoTrackPublication: it represents the publication of a VideoTrack.
  • DataTrackPublication: it represents the publication of DataTrack.
Programmable Video SDK TrackPublication Types.

There are also local and remote subtypes for them:

LocalAudioTrackPublication, LocalVideoTrackPublication and LocalDataTrackPublication represent local Tracks that have been published to the Room by the LocalParticipant. When a Track is published to the Room it gets a SID assigned. That SID can be recovered at the Local TrackPublication object, as the following snippet shows:


_10
localParticipant.publishTrack(localTrack).then(localTrackPublication => {
_10
console.log('Track ${localTrack.name} was published with SID ${localTrackPublication.tracksid}'
_10
})

The LocalParticipant keeps track of all its publications:


_10
localParticipant.tracks.forEach(localTrackPublication => {
_10
console.log('LocalTrackPublication with kind=${localTrackPublication.kind}')
_10
})

Remote TrackPublications

remote-trackpublications page anchor

RemoteAudioTrackPublication, RemoteVideoTrackPublication and RemoteDataTrackPublication are stubs that represent the local TrackPublications of other participants. In other words, a Local TrackPublication on a given Participant will be seen as a Remote TrackPublication on the rest of Participants.


_10
remoteParticipant.on('trackPublished', remoteTrackPublication => {
_10
console.log('A track with sid ${remoteTrackPublication.tracksid} was published to the Room')
_10
})

LocalParticipant and Local Tracks (Client SDK API)

localparticipant-and-local-tracks-client-sdk-api page anchor

To publish a Track developers need to use the publish primitive of the LocalParticipant. When doing so, a new local TrackPublication is created for the specified local Track. Local TrackPublications are owned by the LocalParticipant. Remark also that a Local TrackPublication always have associated one LocalTrack (i.e. the LocalTrack being published) as shown on the following UML diagram:

Programmable Video SDK API - LocalParticipant and Local Tracks.

RemoteParticipants and Remote Tracks

remoteparticipants-and-remote-tracks page anchor

Every Local Track Publication at a LocalParticipant will be seen as a Remote Track Publication at the rest of RemoteParticipants. As a consequence, remote TrackPublication subtypes map to Local TrackPublications.

Note that a Local TrackPublication is always associated to a LocalTrack (i.e. the Track being published). However, Remote TrackPublications behave differently:

  • A Remote TrackPublication has zero Remote Tracks when "this" client is not subscribed to the such published track.
  • A Remote TrackPublication has one Remote Track when "this" client is subscribed to such published Track.
Programmable Video SDK API - RemoteParticipants and Remote Tracks.

Putting Together the Local and the Remote Perspectives (Client SDK API)

putting-together-the-local-and-the-remote-perspectives-client-sdk-api page anchor

Let's use an example to illustrate our API Local and Remote Perspectives. Imagine you create a Group Room using our Room REST API. Also imagine your code creates and distributes the appropriate Access Tokens among participants. In this situation, suppose Alice (client A) and Bob (client B) connect doin the following

  • Alice joins and gets assigned PA_A as her participant SID.
  • Alice publishes a LocalAudioTrack, a LocalVideoTrack and a LocalDataTrack that are assigned SIDs MT_A_A , MT_A_V , and MT_A_D respectively.
  • Alice subscribes to all Tracks.
  • Bob joins and gets assigned PA_B as his participant SID.
  • Bob publishes a LocalAudioTrack, a LocalVideoTrack and a LocalDataTrack that are assigned SIDs MT_B_A , MT_B_V and MT_B_D respectively.
  • Bob subscribes to all Alice's Audio and Data Tracks, but not to her video Tracks.

In this case, the situation will look like the following

Local and Remote Participant example - Twilio Video.

From Alice's perspective (as code executes on Alice's SDK):

  • The LocalParticipant SID is PA_A .
  • The LocalParticipant has three LocalTrackPublications:

    • A LocalAudioTrackPublication with tracksid=MT_A_A . The associated track is a LocalAudioTrack that captures Alice's microphone.
    • A LocalVideoTrackPublication with tracksid=MT_A_V . The associated track is a LocalVideoTrackPublication that captures Alices webcam.
    • A LocalDataTrackPublication with tracksid=MT_A_D . The associated track is a LocalDataTrack where Alice can write her data blobs.
  • There is one RemoteParticipant with SID PA_B .
  • There are three RemoteTrackPublications:

    • A RemoteAudioTrackPublication with tracksid=MT_B_A . As Alice is subscribed to this publication, it has a RemoteAudioTrack instance associated that Alice can use to reproduce Bob's audio.
    • A RemoteVideoTrackPublication with tracksid=MT_B_V . As Alice is subscribed to this publication, it has a RemoteVideoTrack instance associated that Alice can use to display Bob's video.
    • A RemoteDataTrackPublication with tracksid=MT_B_D . As Alice is subscribed to this publication, it has a RemoteDataTrack instance associated that Alice can use to read Bob's data blobs.

From Bob's perspective (as code executes on Bob's SDK):

  • The LocalParticipant SID is PA_B .
  • The LocalParticipant has three LocalTrackPublications:

    • A LocalAudioTrackPublication with tracksid=MT_B_A . The associated track is a LocalAudioTrack that captures Bob's microphone.
    • A LocalVideoTrackPublication with tracksid=MT_B_V . The associated track is a LocalVideoTrack that captures Bob's webcam.
    • A LocalDataTrakPublication with tracksid=MT_B_D . The associated track is a LocalDataTrack where Bob can write his data blobs.
  • There is one RemoteParticipant with SID PA_A .
  • There are three RemoteTrackPublications:

    • A RemoteAudioTrackPublication with tracksid=MT_A_A . As Bob is subscribed to this publication, it has a RemoteTrack instance associated that Alice can use to reproduce Alice's audio.
    • A RemoteVideoTrackPublication with tracksid=MT_A_V . As Bob is not subscribed to this publication, it does not have any associated RemoteVideoTrack instance and cannot display Alice's video.. If Bob got subscribed to this publication, the publication would get that association and would fire a trackSubscribed event to notify it.
    • A RemoteDataTrackPublication with trackSid=MT_A_D . As Bob is subscribed to this publication, it has a RemoteDataTrack instance associated where Bob can read Alice's data blobs.

Rate this page: