Understanding Video Rooms APIs

For the Impatient

This document provides an introduction to Twilio Video Rooms APIs. If you just want to start developing and play with some code you should jump to our Twilio Getting Started Guides:

• Getting Started with Twilio JavaScript SDK
• Getting Started with Twilio Android SDK
• Getting Started with Twilio iOS SDK

Twilio Video Rooms APIs and Interfaces

In the Basic Concepts Guide we explain that Video Rooms applications are split into:

• Application server-side code: that authenticates end-users and contain your business logic.
• Application client-side code: that captures, exchanges and renders the media allowing end-users to communicate in real-time.

Developers can create that code using three Twilio APIs, as the following picture illustrates:

1. The Rooms REST API
2. The Status Callbacks API
3. The Video Client SDK API

The Rooms REST API

The Rooms REST API is based on the following REST resources and dependencies:

Room

A Room represents a virtual space where end-users communicate. Technically, a Room is a computing resource that provides Real-time Communications (RTC) services to client applications through a set of APIs. More specifically, a Room provides:

• A session: so that end-users can connect/disconnect to/from Rooms. When an end-user connects, we say it is a Room Participant.
• An RTC Service: so that Participants can exchange audio, video, and data tracks using WebRTC.

In Twilio, a Room is identified by a SID: a string that uniquely identifies that Room inside Twilio. A Room SID always start with the characters RM and looks like this: RMdba2b037253f3a9a81a591681e88ce96.

Twilio Rooms REST API allows developers to:

• Create new Rooms (POSTing to the /Rooms REST resource)
• List all the Rooms in an Account (GETting the /Rooms REST resource)
• Manage a given Room (POSTing and GETting the /Rooms/{RoomSid} resource)

You can check our Understanding Video Rooms Guide and the Rooms REST API Reference Documentation for more information about Rooms.

Participant

A Participant represents a client (i.e. an end-user) that is connected to a Room and can use the Room’s communication capabilities. In a given Room there can be zero (i.e. an empty Room) or more Participants. A Participant can be connected only to one Room.

In Twilio, a Participant is uniquely identified by a SID with a PA prefix like the following: PA734b33d8bde2a3b3c7720964c87961c9.

Twilio Rooms REST API allows developers to:

• List all the Participants in a Room (GETting the /Participants subresource)
• Manage a given Participant (POSTting and GETting to the /Participants/{ParticipantSid} subresource)

Track

A Track is a stream of bytes that contain the data generated by a multimedia source such as a microphone or a camera. Twilio Rooms are based on a publish/subscribe model. This means that a Participant can publish media tracks to the Room. The rest of Participants can then subscribe to these tracks and start receiving the media information.

In Twilio, a Track is uniquely identified by a SID with the MT prefix like the following: MT6e87a66bd033cdced4efe1267c595a16.

For a given Participant, the Twilio Rooms REST API allows developers to:

• List all the Tracks published by the Participant (GETting the /PublishedTracks subresource of the Participant). In these Tracks the media bytes are sent by the Participant.
• List all the Tracks subscribed by the Participant (GETting the /SubscribedTracks subresource of the Participant). In these Tracks, the media bytes are received by the Participant.

SubscribeRule

A SubscribeRule is a specification of the Tracks a given Participant should subscribe to. For a given Participant, the Twilio Rooms REST API allows developers to:

• Modify the Tracks a Participant is subscribed to (POSTing to the /SubscribeRules subresource of the participant)
• List the SubscribeRules of a Participant (GETting the /SubscribeRules subresource of the Participant)

Using the Rooms REST API

Developers can use the Rooms REST API directly firing the HTTP request to the appropriate endpoints. As an alternative, Twilio has created helper libraries that expose all our REST APIs capabilities in a consistent and seamless way. These are available in the following programming languages: Java, C#, Node.js, Python, Ruby and PHP.

The Status Callbacks API

Status Callbacks allow you to receive events related to the REST resources managed by Twilio. In order to use Status Callbacks you need to configure your Application Server to be able to receive HTTP requests issued by Twilio. Status callbacks events will notify your application when relevant events occur to your Rooms, Participants and Tracks indicating the appropriate SIDs to identify the affected resources. Check the Programmable Video Status Callbacks Guide for further information.

The Video Client SDK API

Video Rooms SDKs are available on three different platforms: JavaScript, Android and iOS. These SDKs share similar type hierarchies based on the following classes: Room, Participant, TrackPublication and Track. The following UML diagram shows their relationship.

Room

This type represent Rooms from the perspective of the client. When a client connects to a Twilio Room it obtains a Room object instance. The Room class has primitives and events that developers can use to:

• Get the Room name and SID
• Get notified when participants connect or disconnect
• Disconnect from the Room.

This snippet shows how to connect to a Room using the JavaScript SDK:

const {connect} = require('twilio-video');
connect('$TOKEN', { name: 'my-room' }).then(room => {
  console.log('I connected to Room ${room.name} that has SID ${room.sid}')
}, error =>{
  console.error('Unable to connect to Room: ${error.message}')
})

Participant

This type represents a Participant from the perspective of the client. A Room can have zero or more Participants.

TrackPublication

A TrackPublication represents the communication of a media Track from a Participant to the Room. A Participant can have zero or more TrackPublications.

Track

This type represents a Track from the perspective of the client.

Understanding Participant Types (Client SDK API)

Our client SDK APIs are designed in “first person”. This means that developers should think about them from the perspective of the client where the code is running. From that perspective, there are two types of Participants as illustrated in the following diagram:

LocalParticipant

It is the Participant on “this” client (i.e. on the client the code is running). The LocalParticipant can access the client hardware resources and manages the status and preferences of the client with respect to the Room. Developers can obtain a reference to the LocalParticipant from the Room instance, as the following code illustrates:

localParticipant = room.localParticipant
console.log('The LocalParticipant identity is ${localparticipant.identity}')
console.log('The LocalParticipant SID is ${localParticipant.sid}')

RemoteParticipant

It represents the Participants of “the rest” of the clients. RemoteParticipant objects can be seen as stubs that represent remote client status and capabilities on “this” client. RemoteParticipants can be obtained from the Room, as the following code illustrates:

//List RemoteParticipants connected to the Room
room.participants.forEach(remoteParticipant => {
  console.log('RemoteParticipant ${remoteParticipant.identity} is connected')
  console.log('RemoteParticipant SID is ${remoteParticipant.sid}')
});

//Get notified when a RemoteParticipant connects to the Room
room.on('participantConnected', remoteParticipant => {
  console.log('RemoteParticipant ${remoteParticipant.identity} just connected')
  console.log('RemoteParticipant SID is ${remoteParticipant.sid}')
});

Understanding Track Types (Client SDK API)

Our SDKs support three Track subtypes:

• AudioTrack: It represents a stream of bytes being generated by an audio source such as a microphone.
• VideoTrack: It represents a stream of bytes being generated by a video source such as a webcam or a screen.
• DataTrack: It represents a stream of bytes (or characters) that are generated by the application. In other words, developers can use a DataTrack to communicate arbitrary data among participants with very low latency. Notice that all Tracks being published by a Participant share the same underlying transport. Hence, the latency properties of the DataTrack bytes published by a Participant should be the the same than the latency properties of the audio and video information published by the Participant.

Following the above mentioned “first person” API mode, each Track subtype has two versions:

Local Tracks

LocalAudioTrack, LocalVideoTrack and LocalDataTrack are the local Track subtypes. Be aware that:

• Instances of these types can only be created at the local SDK.
• Local Tracks do not have a SID because they are not known by Twilio (i.e. they are local to the SDK they have been created in). Tracks are only assigned a SID when they are published.
• Local Tracks have an ID that identifies them locally (i.e. the ID of a published Local Track does not match with the ID of the subscribed Remote Track).
• Local Tracks can be assigned a name at creation time.
• LocalVideoTrack can be attached to the local UI to create a camera preview.

const { createLocalVideoTrack, createLocalAudioTrack } = require('twilio-video');
createLocalAudioTrack({name:'john-audio-track').then(localAudioTrack => {
  console.log('Created LocalAudioTrack with id ${localAudioTrack.id}')
})
createLocalVideoTrack({name:'alices-webcam'}).then(localVideoTrack => {
  console.log('Created LocalVideoTrack with id ${localVideoTrack.id}')
  const localMediaContainer = document.getElementById('local-media');
  localMediaContainer.appendChild(localVideoTrack.attach());
})

Remote Tracks

RemoteAudioTrack, RemoteVideoTrack and RemoteDataTrack are the remote Track subtypes.

• Instances of these types can only exists at the RemoteParticipants that are subscribed to them.
• A Local Track published at a given Participant will be seen as a Remote Track by the rest of subscribed Participants.
• Remote Tracks always have a SID that matches with the SID assigned to the corresponding published Track
• Remote Tracks have a name that matches with the name assigned to the corresponding published Track.
• Remote Tracks are held by the owning RemoteParticipant

remoteParticipant.on('trackSubscribed', remoteTrack => {
    console.log('Received a ${remoteTrack.kind} track with SID ${remoteTrack.sid}')
    document.getElementById('remote-media-div').appendChild(track.attach());
});

Understanding TrackPublication Types (Client SDK API)

As with Track, TrackPublication also has three media-dependent subtypes:

• AudioTrackPublication: it represents the publication of an AudioTrack.
• VideoTrackPublication: it represents the publication of a VideoTrack.
• DataTrackPublication: it represents the publication of DataTrack.

There are also local and remote subtypes for them:

Local TrackPublications

LocalAudioTrackPublication, LocalVideoTrackPublication and LocalDataTrackPublication represent local Tracks that have been published to the Room by the LocalParticipant. When a Track is published to the Room it gets a SID assigned. That SID can be recovered at the Local TrackPublication object, as the following snippet shows:

localParticipant.publishTrack(localTrack).then(localTrackPublication => {
  console.log('Track ${localTrack.name} was published with SID ${localTrackPublication.tracksid}'
})

The LocalParticipant keeps track of all its publications:

localParticipant.tracks.forEach(localTrackPublication => {
  console.log('LocalTrackPublication with kind=${localTrackPublication.kind}')
})

Remote TrackPublications

RemoteAudioTrackPublication, RemoteVideoTrackPublication and RemoteDataTrackPublication are stubs that represent the local TrackPublications of other participants. In other words, a Local TrackPublication on a given Participant will be seen as a Remote TrackPublication on the rest of Participants.

remoteParticipant.on('trackPublished', remoteTrackPublication => {
  console.log('A track with sid ${remoteTrackPublication.tracksid} was published to the Room')
})  

LocalParticipant and Local Tracks (Client SDK API)

To publish a Track developers need to use the publish primitive of the LocalParticipant. When doing so, a new local TrackPublication is created for the specified local Track. Local TrackPublications are owned by the LocalParticipant. Remark also that a Local TrackPublication always have associated one LocalTrack (i.e. the LocalTrack being published) as shown on the following UML diagram:

RemoteParticipants and Remote Tracks

Every Local Track Publication at a LocalParticipant will be seen as a Remote Track Publication at the rest of RemoteParticipants. As a consequence, remote TrackPublication subtypes map to Local TrackPublications.

Note that a Local TrackPublication is always associated to a LocalTrack (i.e. the Track being published). However, Remote TrackPublications behave differently:

• A Remote TrackPublication has zero Remote Tracks when “this” client is not subscribed to the such published track.
• A Remote TrackPublication has one Remote Track when “this” client is subscribed to such published Track.

Putting Together the Local and the Remote Perspectives (Client SDK API)

Let’s use an example to illustrate our API Local and Remote Perspectives. Imagine you create a Group Room using our Room REST API. Also imagine your code also creates and distributes the appropriate Access Tokens among participants. In this situation, suppose Alice (client A) and Bob (client B) connect doin the following

• Alice joins and gets assigned PA_A as her participant SID.
• Alice publishes a LocalAudioTrack, a LocalVideoTrack and a LocalDataTrack that are assigned SIDs MT_A_A, MT_A_V and MT_A_D respectively.
• Alice subscribes to all Tracks.
• Bob joins and gets assigned PA_B as his participant SID.
• Bob publishes a LocalAudioTrack, a LocalVideoTrack and a LocalDataTrack that are assigned SIDs MT_B_A, MT_B_V and MT_B_D respectively.
• Bob subscribes to all Alice’s Audio and Data Tracks, but not to her video Tracks.

In this case, the situation will look like the following

From Alice’s perspective (as code executes on Alice’s SDK):

• The LocalParticipant SID is PA_A.
• The LocalParticipant has three LocalTrackPublications:
  • A LocalAudioTrackPublication with tracksid=MT_A_A. The associated track is a LocalAudioTrack that captures Alice’s microphone.
  • A LocalVideoTrackPublication with tracksid=MT_A_V. The associated track is a LocalVideoTrackPublication that captures Alices webcam.
  • A LocalDataTrackPublication with tracksid=MT_A_D. The associated track is a LocalDataTrack where Alice can write her data blobs.
• There is one RemoteParticipant with SID PA_B.
• There are three RemoteTrackPublications:
  • A RemoteAudioTrackPublication with tracksid=MT_B_A. As Alice is subscribed to this publication, it has a RemoteAudioTrack instance associated that Alice can use to reproduce Bob’s audio.
  • A RemoteVideoTrackPublication with tracksid=MT_B_V. As Alice is subscribed to this publication, it has a RemoteVideoTrack instance associated that Alice can use to display Bob’s video.
  • A RemoteDataTrackPublication with tracksid=MT_B_D. As Alice is subscribed to this publication, it has a RemoteDataTrack instance associated that Alice can use to read Bob’s data blobs.

From Bob’s perspective (as code executes on Bob’s SDK):

• The LocalParticipant SID is PA_B.
• The LocalParticipant has three LocalTrackPublications:
  • A LocalAudioTrackPublication with tracksid=MT_B_A. The associated track is a LocalAudioTrack that captures Bob’s microphone.
  • A LocalVideoTrackPublication with tracksid=MT_B_V. The associated track is a LocalVideoTrack that captures Bob’s webcam.
  • A LocalDataTrakPublication with tracksid=MT_B_D. The associated track is a LocalDataTrack where Bob can write his data blobs.
• There is one RemoteParticipant with SID PA_A.
• There are three RemoteTrackPublications:
  • A RemoteAudioTrackPublication with tracksid=MT_A_A. As Bob is subscribed to this publication, it has a RemoteTrack instance associated that Alice can use to reproduce Alice’s audio.
  • A RemoteVideoTrackPublication with tracksid=MT_A_V. As Bob is not subscribed to this publication, it does not have any associated RemoteVideoTrack instance and cannot display Alice’s video.. If Bob got subscribed to this publication, the publication would get that association and would fire a trackSubscribed event to notify it.
  • A RemoteDataTrackPublication with trackSid=MT_A_D. As Bob is subscribed to this publication, it has a RemoteDataTrack instance associated where Bob can read Alice’s data blobs.