Give Your Video Application an Instant Makeover with the New Virtual Background Library

Time to read: 3 minutes

June 24, 2021
Written by
Donal Toomey
Twilion

Give Your Video Application an Instant Makeover with the New Virtual Background Library

This article is for reference only. We're not onboarding new customers to Programmable Video. Existing customers can continue to use the product until December 5, 2024.


We recommend migrating your application to the API provided by our preferred video partner, Zoom. We've prepared this migration guide to assist you in minimizing any service disruption.

Virtual backgrounds are nothing new in the video conferencing space, with the main out-of-the-box providers offering the ability to insert custom backgrounds and/or blurring. However, historically this functionality has not been easily available to developers who build their own browser-based video applications.

Today, this changes with the availability of a new Video Processor JavaScript library which gives developers the tools to add virtual background support to video applications built on Twilio Video. Virtual backgrounds are currently available in the Chrome and Edge browsers.

GIF image of a participant in a video chat room switching among various virtual backgrounds

Virtual backgrounds can be applied for a variety of use cases, such as for professionals in the financial services, healthcare, education, and customer service spaces, as well as in more playful contexts like social video applications.  

Key benefits

  • Support remote and/or hybrid collaboration: Your end users can work from home and not compromise on their professional look
  • Consistent user experience: Manage the customer experience by using standard or branded backgrounds
  • No need to download another app: Delivers virtual background support within the JavaScript SDK itself, no download or plugin required
  • Streamlined developer toolkit: Simple, easy-to-use API and library that allows for either background blurring or background replacement

Next, we’ll walk you through how you can get up and running quickly with this new library and make it available to your end users.

Video Processor API and Library

The Video Processor API is part of the Twilio Video JavaScript SDK and works on the published video track of the webcam. The video frames are passed to a processor module that can perform the image augmentation and then return the updated frames. The updated frames are then encoded by the Video SDK and sent over the network. Developers can use the new Twilio Video Processor library or write their own video processor code if they want to add on top of background replacement, such as including more playful elements like hats or masks.

The Twilio Video Processor library performs image segmentation and has methods for Gaussian blur and virtual background. Developers can dynamically enable or disable the feature and provide their own background replacement image.

Getting started with the Video Processor Library

The library is available as an NPM package, or you can use the script tag.

NPM

npm install @twilio/video-processors --save

Using this method, you can import twilio-video-processors as follows:

import * as VideoProcessors from '@twilio/video-processors';

Script tag

<script src="https://my-server-path/twilio-video-processors.js"></script>

Using this method, twilio-video-processors.js will set a browser global:

const VideoProcessors = Twilio.VideoProcessors;

The following is an example of using the VirtualBackgroundProcessor class to replace the background of a webcam video stream with a specified background image. The virtual background processor instance is added to the video track as a processor.

import { createLocalVideoTrack } from 'twilio-video';
import { VirtualBackgroundProcessor } from '@twilio/video-processors';

let virtualBackground;
const img = new Image();

img.onload = () => {
  virtualBackground = new VirtualBackgroundProcessor({
    assetsPath: 'https://my-server-path/assets',
    backgroundImage: img,
  });

  virtualBackground.loadModel().then(() => {
    createLocalVideoTrack({
      width: 640,
      height: 480,
      frameRate: 24
    }).then(track => {
      track.addProcessor(virtualBackground);
    });
  });
};
img.src = '/background.jpg';

How it works

The webcam video is passed through a machine learning model that performs a segmentation of the participant’s body from the background. This happens for each video frame. The segmentation results in a bitmap mask for the participant and indicates which pixels are part of the participant and which are part of the background.

The part of the image that contains the person&#x27;s body is highlighted in blue, while the room in the background is unchanged.

Once the mask is available, a blurring effect can be applied to the background or the background can be replaced while leaving the mask unchanged.

The person&#x27;s image appears in front of a background image of green mountains and valleys.

Finally, the updated texture is passed to the browser for encoding and delivery to other participants in the video call. A similar approach is used when a developer-defined background image is provided.

The segmentation machine learning model used in this version of the Video Processor is the MediaPipe Selfie Segmentation model from Google. To optimize the performance for browsers, the model is running on TensorFlowLite and is built for WebAssembly. In addition, the implementation uses the WebAssembly SIMD instruction for performance improvements. This version of the Video Processor library is only supported on the Chrome and Edge browsers.

Try it out today

The Video Processor API was introduced in Twilio Video JavaScript SDK 2.15.0, and the Video Processor JavaScript Library is available today. For additional information, see our developer documentation here. This API and library give developers a new way to customize the user experience when developing Video applications. We can’t wait to see what you build!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.