Work with media frames in Node.js

Video frames use the I420 format. Each frame stores video data in three planes or memory blocks of luminance (Y) set per pixel and chrominance (U, V) set per four-pixel block. Each plane serves as a Buffer. Each also has a stride: the number of bytes per row. The stride spans the plane width at minimum. When rows get padded for alignment, the stride can exceed the plane width.

Each video frame consists of the following parameters:

The luminance plane dimensions get set to the full size of the frame and the chrominance planes get set to half of the frame size.

Input and output use slightly different shapes:

• When you send a frame with the write() method, you pass a flat object with y, u, and v Buffer objects and their yStride, uStride, and vStride values.
• When you receive a frame, each plane arrives wrapped in an object with data, stride, width, and height fields.

The input audio frames carry interleaved 48 kHz mono S16LE PCM samples in a single Buffer. You pass only the pcm buffer and the number of frames.

• S: Use Signed positive or negative integer values.
• 16: Store 16 bits (or two bytes) of data per audio sample.
• LE: Use the little-endian method to store data placing the least significant byte in the smallest memory address.
• PCM: Convert data using Pulse-code modulation: the raw, uncompressed audio wave data.

The output audio frames can vary. Each frame reports its own sampleRate, channels, and frames, along with the pcm buffer and a bigint timestampNs.

Until you connect to a Room, you can't access a track source. When the source isn't ready, the write() method returns false and throws an error on invalid input. After the connect() method returns a response, start your send loop.

To publish video, create a local video track and call the write() method for each I420 frame.

1
const { createLocalVideoTrack } = require('@twilio/video-node-sdk');
2

3
const videoTrack = createLocalVideoTrack('virtual-camera');
4
// Pass videoTrack to connect() or publish it later.
5

6
videoTrack.write({
7
  y: yPlane,
8
  u: uPlane,
9
  v: vPlane,
10
  yStride: 1280,
11
  uStride: 640,
12
  vStride: 640,
13
  width: 1280,
14
  height: 720,
15
  timestampNs: process.hrtime.bigint(), // optional; defaults to now
16
});

A real app loops write() method calls at the source frame rate.

To publish audio, create an audio track and call the write() method for the 48 kHz mono PCM buffer of a certain number of frames:

Register a callback with onFrame() on a subscribed remote track.

• With video, each plane arrives as an object with a data buffer and a stride.
• With audio, each frame arrives as an object with four properties.

1
function trackSubscribed(track) {
2
  if (track.kind === 'video') {
3
    track.onFrame(frame => {
4
      const { width, height } = frame;
5
      const yData = frame.y.data;
6
      const yStride = frame.y.stride;
7
      // Process the frame, then return.
8
    });
9
  }
10

11
  if (track.kind === 'audio') {
12
    track.onFrame(frame => {
13
      // frame.pcm, frame.sampleRate, frame.channels, frame.frames
14
    });
15
  }
16
}

To stop receiving, call removeFrameCallback() on the track.

The following example comes from the video_mirror.js example in the SDK repository. This code receives remote video and sends it straight back into the Room, mapping each received plane onto a write() input.

Send frames at their real frame rate and keep timestamps moving forward.

Audio has extreme time-sensitivity. If you write frames on a plain setInterval, playback might drift and click. Pace the playback using a drift-compensated writer that drains a buffer queue at exactly 48 kHz.

To review a working example, see audio_push.js and its helpers/paced-audio-writer.js in the SDK repository.

• Best practices: Pace frames, manage memory, and troubleshoot common issues.
• API Reference: Browse the full frame and track APIs.