Work with media frames in Node.js

(new)

Public Beta

The Video Media SDK for Node.js is currently available as a Public Beta product and the information contained in this document is subject to change. This means that some features are not yet implemented and others may be changed before the product is declared as Generally Available. Public Beta products are not covered by the Twilio Support Terms or Twilio Service Level Agreement.

The Video Media SDK for Node.js is not a US Health Insurance Portability and Accountability Act (HIPAA) Eligible Service or Payment Card Industry Data Security Standard (PCI DSS) compliant and should not be enabled in workflows that are subject to HIPAA or PCI.

The Video Media SDK for Node.js works with media one frame at a time. A frame represents one unit of decoded media. You can send either one frame of video as one picture in I420 format or one frame of audio frame as a sample of sound.

Using the SDK, write code that sends frames into a Room with the write() method and receives frames with the frames() async iterator.

Video frames

Video frames use the I420 format. Each frame stores video data in three planes: luminance (Y) at full resolution, and two chrominance planes (U, V) at half resolution in each dimension. Each plane is an object that holds the plane's data as a Buffer, its stride, and its own width and height. The stride is the number of bytes per row. It's at least the plane width, and it can exceed the width when rows are padded for alignment.

Video frame data parameters

Each video frame consists of the following parameters:

Parameter	Type	Necessity	Accepted values
`width`	integer	Required	Frame width in pixels. Must be positive and even.
`height`	integer	Required	Frame height in pixels. Must be positive and even.
`y`	plane object	Required	Luminance plane
`u`	plane object	Required	Blue-difference chrominance (`Cb`) plane
`v`	plane object	Required	Red-difference chrominance (`Cr`) plane
`format`	string	Optional	`'I420'`, the only accepted value
`timestamp`	number	Optional	Presentation time in microseconds. Defaults to the current time.
`rotation`	integer	Optional	Degrees of rotation expressed as one of four values: `0`, `90`, `180`, `270`.

Each plane object has the following fields:

Field	Type	Description
`data`	Buffer	The plane's samples, at least `stride` × `height` bytes long.
`stride`	integer	Bytes per row. At least `width`, and possibly padded for alignment.
`width`	integer	Plane width in samples. Half the frame width for `U` and `V`.
`height`	integer	Plane height in samples. Half the frame height for `U` and `V`.

Video frame plane sizes

The luminance plane dimensions get set to the full size of the frame and the chrominance planes get set to half of the frame size.

Plane	Logical size	Buffer size	Description
`Y`	`width` × `height`	`y.stride` × `height`	Luminance
`U`	`⌈width/2⌉` × `⌈height/2⌉`	`u.stride` × `⌈height/2⌉`	Blue-difference chrominance (`Cb`)
`V`	`⌈width/2⌉` × `⌈height/2⌉`	`v.stride` × `⌈height/2⌉`	Red-difference chrominance (`Cr`)

Sending and receiving use the same shape. Each of y, u, and v is a plane object, so you can write a received frame straight back out without reshaping it.

Audio frames

The input audio frames carry interleaved 48 kHz mono S16LE PCM samples in a single Buffer. You pass only the pcm buffer and the number of frames.

S: Use Signed positive or negative integer values.
16: Store 16 bits (or two bytes) of data per audio sample.
LE: Use the little-endian method to store data placing the least significant byte in the smallest memory address.
PCM: Convert data using Pulse-code modulation: the raw, uncompressed audio wave data.

The output audio frames can vary. Each frame reports its own sampleRate, channels, and frames, along with the pcm buffer and a timestamp in microseconds.

Send frames into a Room

The write() method returns false when the SDK doesn't send a frame. For video, that most often means you wrote the frame before connect() resolved. For audio, it means the frame didn't fit in the publish queue. The method throws a TypeError or RangeError exception on invalid input. Start your send loop after connect() resolves.

Send video frames

To publish video, create a local video track and call the write() method for each I420 frame.

Send one video frame

1const { createLocalVideoTrack } = require('@twilio/video-node-sdk');
2
3const videoTrack = createLocalVideoTrack('virtual-camera');
4// Pass videoTrack to connect() or publish it later.
5
6videoTrack.write({
7  format: 'I420',
8  width: 1280,
9  height: 720,
10  y: { data: yPlane, stride: 1280, width: 1280, height: 720 },
11  u: { data: uPlane, stride: 640, width: 640, height: 360 },
12  v: { data: vPlane, stride: 640, width: 640, height: 360 },
13  // timestamp is optional; it defaults to the current time in microseconds.
14});

A real app loops write() method calls at the source frame rate.

Send audio frames

To publish audio, create an audio track and call the write() method for the 48 kHz mono PCM buffer of a certain number of frames:

Send a series of audio frames

1const { createLocalAudioTrack } = require('@twilio/video-node-sdk');
2
3const audioTrack = createLocalAudioTrack('mic');
4
5audioTrack.write({
6  pcm: pcmBuffer, // interleaved int16 samples
7  frames: 480, // samples in this buffer
8});

Receive frames from a Room

Read frames from a subscribed remote track with the frames() async iterator.

With video, each plane arrives as a plane object with data, stride, width, and height.
With audio, each frame carries its pcm buffer, sampleRate, channels, and frames, along with its timestamp.

Receive video or audio frames

1async function trackSubscribed(track) {
2  if (track.kind === 'video') {
3    for await (const frame of track.frames()) {
4      const { width, height } = frame;
5      const yData = frame.y.data;
6      const yStride = frame.y.stride;
7      // Process the frame.
8      frame.close?.();
9    }
10  } else if (track.kind === 'audio') {
11    for await (const frame of track.frames()) {
12      // frame.pcm, frame.sampleRate, frame.channels, frame.frames
13      frame.close?.();
14    }
15  }
16}

The loop ends when the track is unsubscribed or the Room disconnects. To stop receiving sooner, exit the loop with a break statement.

The following example comes from the video_mirror.js example in the SDK repository. This code receives remote video and sends it back into the Room by passing each received frame's planes to write().

Remote video sent to a room

1for await (const frame of track.frames()) {
2  videoTrack.write({
3    format: 'I420',
4    width: frame.width,
5    height: frame.height,
6    y: frame.y,
7    u: frame.u,
8    v: frame.v,
9    timestamp: frame.timestamp,
10    rotation: frame.rotation,
11  });
12  frame.close?.();
13}

Frame timing and pacing

Send frames at their real frame rate and keep timestamps moving forward.

Audio has extreme time-sensitivity. If you write frames on a plain setInterval, playback might drift and click. Pace the playback using a drift-compensated writer that drains a buffer queue at exactly 48 kHz.

To review a working example, see audio_push.js and its helpers/paced-audio-writer.js in the SDK repository.

Next steps

Best practices: Pace frames, manage memory, and troubleshoot common issues.
API Reference: Browse the full frame and track APIs.