Announcing Programmable Video Network Quality API

When the network is the problem, what’s the solution?

Most of us have experienced the annoying effects of degraded video conferencing quality: audio becomes robotic and choppy while video turns into a sequence of blurry pictures. This is frustrating for end-users who tend to complain to application developers and platform providers asking for a solution. However, experience shows that most of the time, what causes these problems is the network access of the end users themselves. Notably, crappy wifi.

Wireless links are subject to so many different types of interference that can cause packet loss that it’s impossible to control. Worse, the use of shared access mechanisms split the available bandwidth amongst channels and may cause bitrate drops in crowded areas. Even with wired ethernet, routers and firewalls might get congested causing jitter, latency and packet loss. Worst of all, free, coffee shop wifi.

When video conferencing, all these problems contribute to the degradation of the perceived quality.

Whilst you can’t fix every wifi network, you can tell users that they’re on a crappy network by sharing network quality data:

• End-users will understand why they are suffering degraded quality and will be able to identify the actual root cause. This avoids tons of frustration and minimizes complaints to application developers.
• End-users may be able to improve the situation by selecting a different network or getting closer to their access point.

However, it is hard for application developers to evaluate the network quality and to determine when those notifications should be generated, and what information they should contain. Twilio Programmable Video Network Quality API provides a simple solution to this problem.

Evaluating Network Quality: the old way.

If you are a WebRTC developer you may have heard about the WebRTC Stats: a candidate (at the time of this writing) W3C recommendation that defines a set of quality-related metrics. These include packet loss, jitter, round trip time, bitrate, nack count and many others.

Collecting such stats in your application is simple. However, understanding how they influence quality is tricky and requires expertise and significant code. Just to illustrate, have a look at the following WebRTC Stats samples.

• {availableIncomingBitrate: 1234732, availableOutgoingBitrate: 1198786, bytesReceived: 12680339, bytesSent: 26676338, consentRequestsSent: 70, currentRoundTripTime: 0.206, id: "RTCIceCandidatePair_9l7fLSCK_QdSCc/ua", localCandidateId: "RTCIceCandidate_9l7fLSCK", nominated: true, priority: 7961835275829642000, remoteCandidateId: "RTCIceCandidate_QdSCc/ua", requestsReceived: 11, requestsSent: 1, responsesReceived: 67, responsesSent: 11, state: "in-progress", timestamp: 1537536665432.738, totalRoundTripTime: 10.01, transportId: "RTCTransport_audio_1", type: “candidate-pair"}
• {bytesReceived: 677062, codecId: "RTCCodec_audio_Inbound_111", fractionLost: 0.0390625, id: "RTCInboundRTPAudioStream_1470361167", isRemote: false, jitter: 0.005, mediaType: "audio", packetsLost: 174, packetsReceived: 7821, ssrc: 1470361167, timestamp: 1537536665432.738, trackId: "RTCMediaStreamTrack_receiver_1", transportId: "RTCTransport_audio_1", type: “inbound-rtp"}
• {bytesSent: 715579, codecId: "RTCCodec_audio_Outbound_111", id: "RTCOutboundRTPAudioStream_2926639135", isRemote: false, mediaType: "audio", packetsSent: 8143, ssrc: 2926639135, timestamp: 1537536665432.738, trackId: "RTCMediaStreamTrack_sender_1", transportId: "RTCTransport_audio_1", type: “outbound-rtp"}
• {bytesReceived: 11475261, codecId: "RTCCodec_video_Inbound_96", firCount: 0, fractionLost: 0.09765625, framesDecoded: 3809, id: "RTCInboundRTPVideoStream_267864872", isRemote: false, mediaType: "video", nackCount: 211, packetsLost: 109, packetsReceived: 11754, pliCount: 2, qpSum: 92461, ssrc: 267864872, timestamp: 1537536665432.738, trackId: "RTCMediaStreamTrack_receiver_2", transportId: "RTCTransport_audio_1", type: “inbound-rtp}
• {bytesSent: 25399718, codecId: "RTCCodec_video_Outbound_96", firCount: 84, framesEncoded: 11056, id: "RTCOutboundRTPVideoStream_3543327014", isRemote: false, mediaType: "video", nackCount: 721, packetsSent: 29798, pliCount: 24, qpSum: 312845, ssrc: 3543327014, timestamp: 1537536665432.738, trackId: "RTCMediaStreamTrack_sender_2", transportId: "RTCTransport_audio_1", type: “outbound-rtp"}

Now try to figure out the quality of the associated network link. Is the network causing a degradation on the perceived quality of experience? What could you tell the end-user in relation to her network link?

WebRTC Stats is a powerful tool in the hands of experts but non-trivial to communicate to end-users meaningful information about how their networks are affecting the quality of their call.

Evaluating Network Quality with the Network Quality API

Every WebRTC developer should not have to wrangle WebRTC Stats to understand end-user video quality. To solve this Twilio has created the Network Quality API for Group Rooms, a simple-to-understand Network Quality Level on a Likert scale from 5 (very good network) to 1 (very bad network). It uses a complex algorithm that ingests network data and computes a score every ~2 seconds.

You can visually depict quality level in your applications using the ubiquitous “signal strength indicator” from mobile phones. This provides an intuitive way of communicating to end-users the quality of their networks without requiring explanation.

It is important to note that the Network Quality API evaluates the network quality and not the perceived quality. Hence a quality level of 5 does not necessarily mean that the audio and video quality are good because the perceived quality may be degraded for reasons other than the network (e.g. CPU starvation, misconfiguration, etc.) In general, when the Quality Level goes down, the perceived quality will also go down. Hence, the Network Quality API can be a powerful tool for diagnosing whether the network is the root cause of audio and video quality degradation.

The following table illustrates an approximate guide to perceived quality:

Getting started with the Network Quality API

The Network Quality API is currently available on both Small Group Rooms and Group Rooms using the Twilio Programmable Video JavaScript SDK 1.14+. Using it is easy, you just need to:

• Activate it on when your Local Participant connects to a Room.
• Subscribe to the Quality Level events on that Local Participant.

The following code snippet illustrates how to do it:

//Activate Network Quality when connecting to a room
Video.connect(token, {
   //... all your connect parameters here
  networkQuality: true
})

//Subscribe to the Network Quality Level
localParticipant.on('networkQualityLevelChanged', myCallbackFunction);

For further details, take a look at the Network Quality API official documentation.

What’s next with the Network Quality API

This is just the beginning. We’re planning to ship:

• Android and iOS SDK support
• Sharing the quality level for all the participants (not only the local participant)
• Detailed information on what’s the root cause of a degraded quality level (e.g. inbound/outbound direction, specific audio/video stats, etc.)

If you are missing something or you think we can improve the Network Quality API do not hesitate to reach us.

We can’t wait to see what you build!