An AudioProcessor can be added to the SDK, providing access to the audio input stream and the ability to process or analyze the stream before sending it to Twilio. To add the processor, you must implement the AudioProcessor interface and use device.audio.addProcessor. You can use device.audio.removeProcessor to remove it.
Use cases include:
Example:
The following example demonstrates how to utilize AudioProcessor APIs to use background music for local audio instead of using a microphone.
_47import { AudioProcessor, Device } from '@twilio/voice-sdk';_47_47let audioContext;_47_47class BackgroundAudioProcessor implements AudioProcessor {_47_47 private audioContext: AudioContext;_47 private background: MediaElementAudioSourceNode;_47 private destination: MediaStreamAudioDestinationNode;_47_47 constructor() {_47 if (!audioContext) {_47 audioContext = new AudioContext();_47 }_47 this.audioContext = audioContext;_47 }_47_47 async createProcessedStream(stream: MediaStream): Promise<MediaStream> {_47 // Create the source node_47 const audioEl = new Audio('/background.mp3');_47 audioEl.addEventListener('canplaythrough', () => audioEl.play());_47 this.background = this.audioContext.createMediaElementSource(audioEl);_47_47 // Create the destination node and connect the source node_47 this.destination = this.audioContext.createMediaStreamDestination();_47 this.background.connect(this.destination);_47_47 // Return the resulting MediaStream_47 return this.destination.stream;_47 }_47_47 async destroyProcessedStream(stream: MediaStream): Promise<void> {_47 // Cleanup_47 this.background.disconnect();_47 this.destination.disconnect();_47 }_47}_47// Construct a device object, passing your own token and desired options_47const device = new Device(token, options);_47_47// Construct the AudioProcessor_47const processor = new BackgroundAudioProcessor();_47_47// Add the processor_47await device.audio.addProcessor(processor);_47// Or remove it later_47// await device.audio.removeProcessor(processor);
Called by the SDK whenever the active input audio stream is updated. Use this method to initiate your audio processing pipeline and return the resulting audio stream in a Promise<MediaStream>.
This method has one argument which represents the current input audio stream. This is the MediaStream object from the input device, such as the microphone. You can process or analyze this stream and create a new stream that will be sent to Twilio.
Called by the SDK after the original input audio stream and the processed stream has been destroyed. The stream is considered destroyed when all of its tracks are stopped and its references in the SDK are removed. This method is called whenever the current input stream is updated. Use this method to run any necessary teardown routines needed by your audio processing pipeline and return a Promise<void> representing the result of the teardown process.
This method has one argument which represents the destroyed processed audio stream.