Lee Martin Gets You To Focus On One Thing Using ThreeJS, HTML and Your Own Face

Time to read: 2 minutes

December 04, 2017
Written by
Kyle Kelly-Yahner
Twilion

SouthernGothic

This article is for reference only. We're not onboarding new customers to Programmable Video. Existing customers can continue to use the product until December 5, 2024.


We recommend migrating your application to the API provided by our preferred video partner, Zoom. We've prepared this migration guide to assist you in minimizing any service disruption.

Lee Martin
Maybe you’re more likely to focus on one thing if that thing is your own face.

That’s the general philosophy behind Lee Martin’s latest work. He’s battling against our insatiable appetite for internet distraction in hopes that you’ll focus on music, and just music, for a few moments.

Lee built out a website for 14-time Grammy Award Winner, Dan Tyminski’s new album, Southern Gothic, that mirrors Dan’s music and also your face — literally.
 

Lee’s creation, The Garden of Good and Evil, is a world made in the style of Dan’s album. Picture frames hang from a black tree’s branches as Dan’s new album plays. As the branches rotate, you see a real-time video (powered by Twilio) of your face in those branches (assuming you let the site access your camera).

Holding Up A Mirror

“Seeing the impact from the listening events and latching onto Dan’s statement of ‘holding up a mirror to society,’ I proposed a concept that would cast the filtered faces of real-time listeners into a gothic scene while allowing users the chance to stream tracks from the new album,” says Lee.

At a cursory glance, Lee thinks of the site as a nod to Turntable.fm with a dash of Chat Roulette. But, Lee’s code and the craftsmanship therein show that this site is more than a nod to a predecessor. This is how Lee built it.

The Code That Gets Your Face Onto A Gothic Tree

    • Lee uses Twilio Video to display users’ faces in each picture frame and display multiple users on multiple tree branches.
    • To crop the video size to match each picture frame he StackOverflowed his way to this block of code

 

video.onloadedmetadata = function(){
 // this.videoHeight;
 // this.videoWidth;
}

 

  • To blend users’ videos along with the album artwork, Lee used an HTML5 canvases and the drawImage module:

     

var canvas    = document.createElement('canvas');
canvas.height = 128;
canvas.width  = 128;
var context   = canvas.getContext('2d');
function render() {
 context.drawImage(userVideo, 0, 0, 128, 128);
 context.globalCompositeOperation = 'multiply';
 context.drawImage(textureVideo, 0, 0, 128, 128);
 requestAnimationFrame();
}

 

    • To render the video in 3D, place the video on the correct plane, (without making anyone’s device catch on fire) Lee relied on ThreeJS. He heavily used on OBJLoader.

 

var geometry = new THREE.PlaneGeometry(60, 80, 1, 1);
var texture = new THREE.Texture(canvas);
texture.repeat.x = 96 / 128;
texture.offset.x = 0.125;
var material = new THREE.MeshBasicMaterial({
 map: texture
});
var plane = new THREE.Mesh(geometry, material);
// position your plane

 

Sudo Git Attention

Rendering real-time video on a 3-D HTML5 canvas is no small feat. But, getting someone to focus solely on music as an event might be even harder. Lee tackled both challenges at once. Engineering your way around our built-in distraction machines is incredibly difficult, but to Lee it’s worth it.

“We can’t forget that recorded music is art and while we will consume it personally from our pocket, it should have a grand opening as well.”

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.