Create A Family Photo Frame With Twilio, AWS, and Electric Objects

Time to read: 13 minutes

September 29, 2016
Written by
Jed Schmidt
Contributor
Opinions expressed by Twilio contributors are their own

jedfeatured

Jed Schmidt is a developer, dad, musician and all around rad dude working in New York. He graces the Twilio blog with his presence and a lovely tutorial on using Twilio, AWS and Electric Objects to create a Twilio MMS photo frame. You can watch him build the whole thing in the following video tutorial, and follow along with him in his detailed step by step instructions. Soon you’ll have a code-powered photo collage going in no time.

We’ll let Jed take it from here.

Using Twilio, AWS, and Electric Objects to create an MMS-powered family photo frame.

A few months ago my partner and I welcomed a pudgy baby boy named Ko into our lives. Since then it’s been amazing to see the small changes in him day by day, like when he discovered his hands, or learned how to smile. So I’ve found myself taking out my phone a lot more often, to capture as many of these moments as I can.

And of course the demands from my parents for pictures of their new grandson have been fierce. Ordinarily I’d post things like this on Facebook, but having been on the receiving end of oversharing parents there for a while now, I wanted something a little less public. Sure, I could deal with Facebook’s ever-changing twiddly permissions UI to make sure photos only went to my parents, but since we were already using iMessage to communicate, I decided to send them pictures there.

This was definitely the path of least resistance, but of course, the iMessage UI isn’t really a great way to get a snapshot of someone’s most recent pictures. So I decided to get creative, and turn my Electric Objects Digital Art Display into a collaborative family photo frame, so that my parents could see new pictures show up in their kitchen, in real time. I’ve used digital picture frames like Ceiva in the past, and they’re okay, but nothing matches the sleek finish of the Electric Objects display, or the ability to control how images are loaded and displayed. And if you get bored of family photos you can always use it to show Jenn Schiffer‘s excellent pixel art or follow Tom MacWright‘s lead and make your own animated art.

frame
Having seen a bunch of great SMS/MMS demos from Ricky Robinett at BrooklynJS, I figured Twilio would be an easy way to pull all of the photos out of our existing iMessage group and put them on the display. So I created an app that uses AWS to glue Twilio and Electric Objects together, and put the source code on GitHub. Here’s how the whole thing works:

  1. I created a new Twilio phone number and added it to the existing iMessage group my family uses to communicate.
  2. Twilio sends all messages that arrive at this phone number to a Lambda function, via API Gateway.
  3. The function copies any MMS images from Twilio to an S3 bucket, then pulls the most recent images from the bucket to compose a collage, which is then saved back to the bucket.
  4. The function then tells Electric Objects to update the contents of my parent’s display to the URL of the collage image.

From my phone to their display, the whole process takes about 15 seconds, a nice side effect of which is I get a backup of all the photos in an S3 bucket.

Creating Your Own Collaborative Family Photo Frame

In this post, I’m going to show you how to create your own collaborative family photo frame. All you need to follow along is:

Once you’ve got those sorted, now you’ll just need to:

  1. walk through the source code (optional, only if you’re interested!), and then
  2. follow the setup instructions for AWS and Twilio.

If you’re stuck trying to get your frame set up, please feel free to ask  for help in the comments or tweet us!

Diving Into The Code

This is a walk-through of the AWS Lambda code used to build the MMS-powered family photo frame described in this post. You don’t need this to create your own frame, but it’s always good to understand the code you’ll be running. This document was generated from the source using docco.

First, let’s bring in some libraries to do the heavy lifting.

We’ll want to parse querystring payloads, make file system and process calls promise-friendly, fetch MMS image payloads from Twilio over HTTPS, store and retrieve images to and from S3, resize and compose images with ImageMagick, crop the most interesting square in an image, and finally, update our frame.

var qs = require('querystring')
var fs = require('mz/fs')
var child = require('mz/child_process')
var got = require('got')
var aws = require('aws-sdk')
var gm = require('gm')
var smartcrop = require('smartcrop-gm')
var eo = require('electric-objects')

AWS Lambda ships with ImageMagick, so we’ll use that.

var im = gm.subClass({imageMagick: true})

We’ll be reusing the same client for all S3 calls.

var s3 = new aws.S3()

We’ll store the name of our frame globally for easier access.

var frameName

Now, let’s define the layout of the collage.

var layout = []

Since the aspect ratio of the EO1 frame is 9:16, we’ll define our layout as a set of 16 squares in decreasing size from 6×6 to 1×1, and place them carefully to fill every pixel, as shown in the image below. Many thanks to Nikki Sylianteng for helping me find a layout that’s easy on the eyes!

layout 
layout[ 0] = {size: 6, left: 0, top:  0}

layout[ 1] = {size: 5, left: 4, top:  8}

layout[ 2] = {size: 4, left: 0, top:  6}
layout[ 3] = {size: 4, left: 0, top: 12}

layout[ 4] = {size: 3, left: 6, top:  0}
layout[ 5] = {size: 3, left: 6, top:  5}
layout[ 6] = {size: 3, left: 6, top: 13}

layout[ 7] = {size: 2, left: 7, top:  3}
layout[ 8] = {size: 2, left: 4, top:  6}
layout[ 9] = {size: 2, left: 0, top: 10}
layout[10] = {size: 2, left: 2, top: 10}
layout[11] = {size: 2, left: 4, top: 14}

layout[12] = {size: 1, left: 6, top:  3}
layout[13] = {size: 1, left: 6, top:  4}
layout[14] = {size: 1, left: 4, top: 13}
layout[15] = {size: 1, left: 5, top: 13}

Now we export our handler, the function that’s called whenever our AWS Lambda function is invoked. We’ll get the frame name from the function name, handle all rejected promises globally to simplify our code, and then pass the message to our onmessage listener.

exports.handler = function(message, context) {
  frameName = context.functionName

  process.once('unhandledRejection', context.fail)

  onmessage(message).then(context.succeed, context.fail)
}

When a message arrives, we’ll parse it into images. If no images exist, we’ll return early, otherwise we’ll handle each image and then create the collage.

function onmessage(message) {
  var images = parse(message)

  if (images.length < 1) return

  return Promise.all(images.map(onimage)).then(onimages)
}

Let’s pull the images out of the payload from the Twilio webhook. Identifiers are created by subtracting the timestamp and sequence number from an arbitrarily large integer, so that newer images will sort lexicographically before older images, allowing us to query S3 for only the most recent ones, without iterating through the whole bucket.

function parse(message) {
  var data = qs.parse(message)
  var now = Date.now()

  return Array.from({length: data.NumMedia}).map((n, seq) => {
    var url = data[`MediaUrl${seq}`]
    var contentType = data[`MediaContentType${seq}`]
    var id = (Number.MAX_SAFE_INTEGER - now - seq).toString(36)
    var suffix = contentType.split('/').pop()
    var path = `images/${id}.${suffix}`
    var message = data.Body

    return {url, contentType, path, message}
  })
}

For each incoming image, we’ll need to fetch it from Twilio, crop it to a square, and then upload it to our S3 bucket.

function onimage(image) {
  return Promise.resolve(image)
    .then(fetch)
    .then(crop)
    .then(upload)
}

We’ll pull the image as a buffer from its Twilio URL.

function fetch(image) {
  return got.get(image.url, {encoding: null}).then(res => {
    return Object.assign(image, {body: res.body})
  })
}

We’ll crop the most interesting square out of the image, resize it to 1080 pixels (the width of the display), and update the image body.

function crop(image) {
  var size = {width: 1080, height: 1080}

  return smartcrop.crop(image.body, size).then(data => {
    var w = data.topCrop.width
    var h = data.topCrop.height
    var x = data.topCrop.x
    var y = data.topCrop.y

    return new Promise((resolve, reject) => {
      im(image.body).crop(w, h, x, y).toBuffer((err, body) => {
        err ? reject(err) : resolve(Object.assign(image, {body}))
      })
    })
  })
}

We’ll upload the image to S3.

function upload(image) {
  var params = {
    Bucket: frameName,
    Key: image.path,
    Body: image.body,
    ContentType: image.contentType,
    Metadata: {message: image.message}
  }

  return s3.putObject(params).promise()
}

Once all of the inbound images are processed, we’ll need to list and then download the latest ones, compose them into the collage, publish it to S3, and then tell the frame that the collage has been updated.

function onimages() {
  return Promise.resolve()
    .then(list)
    .then(download)
    .then(compose)
    .then(publish)
    .then(update)
}

Let’s list the keys of the 16 most recent images in our S3 bucket.

function list() {
  var listObjects = s3.listObjectsV2({
    Bucket: frameName,
    Delimiter: '/',
    MaxKeys: 16,
    Prefix: 'images/',
    StartAfter: 'images/'
  }).promise()

  return listObjects.then(res => res.Contents.map(item => item.Key))
}

Now let’s download all the images we need to /tmp/images, creating the directory if it doesn’t already exist.

function download(keys) {
  return fs.mkdir('/tmp/images').catch(() => {}).then(() => {
    var downloads = keys.map(Key => {
      return new Promise(resolve => {
        var getObject = s3.getObject({Bucket: frameName, Key})
        var rs = getObject.createReadStream()
        var ws = fs.createWriteStream(`/tmp/${Key}`)
        rs.pipe(ws)
        ws.on('finish', resolve)
      })
    })

    return Promise.all(downloads).then(() => keys)
  })
}

Let’s compose the collage with ImageMagick, and then save it to /tmp. We’ll use a black background by default for missing images, and set the dimensions to 1080p portrait to match those of the frame.

function compose(keys) {
  var cmd = 'convert -size 1080x1920 xc:black'

  keys.forEach((key, i) => {
    var xy = `${layout[i].left * 120},${layout[i].top * 120}`
    var wh = `${layout[i].size * 120},${layout[i].size * 120}`
    cmd += ` -draw "image Over ${xy} ${wh} '/tmp/${key}'"`
  })

  cmd += ' /tmp/collage.jpeg'

  return child.exec(cmd)
}

Once the collage is done, we’ll publish it on S3 so that our frame can reach it.

function publish() {
  var params = {
    Bucket: frameName,
    Key: 'collage.jpeg',
    Body: fs.createReadStream('/tmp/collage.jpeg'),
    ContentType: 'image/jpeg',
    ACL: 'public-read'
  }

  return new Promise((resolve, reject) => {
    var cb = err => err ? reject(err) : resolve()

    s3.upload(params, cb)
  })
}

To update our frame, we’ll need to pull the credentials from S3 and use them to log in to the Electric Objects website, and use their set_url page to point the frame to the collage URL.

function update() {
  var url = `http://s3.amazonaws.com/${frameName}/collage.jpeg`
  var params = {Bucket: frameName, Key: 'eo-config.json'}
  var getObject = s3.getObject(params).promise()

  return getObject.then(res => {
    var account = JSON.parse(res.Body.toString('utf8'))
    var client = eo(account.email, account.password)

    return client.setUrl(url).then(() => url)
  })
}

Setting Up Twilio, Amazon Lambda and API Gateway

All in all this setup should take you about ten minutes. Setup is divided into four parts:

  1. Download the Lambda code and Electric Objects login information,
  2. set up an S3 bucket, IAM role, Lambda function, and API Gateway endpoint on AWS,
  3. set up a Twilio phone number, and point it at your API Gateway endpoint,
  4. send an MMS to your number to test that it works.

You can follow along to this video to complete this steps, or follow the written instructions below:

Download the lambda code

Download the Lambda code from GitHub, and save it anywhere on your computer where you can find it again.

Download and edit the Electric Objects login JSON file

  1. Download Electric Objects login JSON file from GitHub, and save it in the same location as above.
  2. Open this file in your favorite text editor, replace YOUR-ELECTRIC-OBJECTS-EMAIL-ADDRESS and YOUR-ELECTRIC-OBJECTS-PASSWORD with your Electric Objects email address and password, and then save your changes.

Create An AWS S3 Bucket

This is where you’ll store your EO1 account settings, the images coming in from Twilio, and the composited collage to be sent to the Electric Objects display.

  1. From the top nav of the AWS Console, choose Services, then All AWS Services, then S3.
  2. Click the Create bucket button.
  3. For Bucket Name, specify the name of your project. Here, mine is jed-family-frame.
  4. For Region, choose US Standard.
  5. Click the Create button.
  6. Click the Upload button.
  7. Click the Add Files button.
  8. Choose the eo-config.json file downloaded in the previous step.
  9. Click the Start Upload button.
  10. Your bucket is ready.

Create an AWS IAM role

This gives your Lambda function the permissions it needs to read from and write to the S3 bucket.

  1. From the top nav of the AWS Console, choose Services, then All AWS Services, then IAM.
  2. In the left sidebar, click the Roles button.
  3. Click the Create New Role button.
  4. For Role Name, specify the same name of your project as you did for the S3 bucket.
  5. Click the Select button for AWS Lambda, under AWS Service Roles.
  6. Select the checkboxes next to AmazonS3FullAccess and CloudWatchLogsFullAccess.
  7. Click the Next Step button.
  8. Click the Create Role button.
  9. Your role is ready.

Create an AWS Lambda function

  1. From the top nav of the AWS Console, choose Services, then All AWS Services, then Lambda.
  2. Click the Skip button.
  3. Click the Next button.
  4. For Name, specify the same name of your project as you did for the S3 bucket.
  5. For Runtime, choose Node.js 4.3.
  6. For Code entry type, choose Upload a .ZIP file.
  7. Click the Upload button and choose the lambda.zip file downloaded earlier.
  8. For Handler, specify index.handler.
  9. For Role, choose Choose an existing role.
  10. For Existing role, choose the name of the role you created.
  11. For Memory (MB), specify 1024.
  12. For Timeout, specify 1 min.
  13. For VPC, choose No VPC.
  14. Click the Next button.
  15. Click the Create function button.
  16. Your lambda is ready.

Create an AWS API Gateway endpoint

  1. From the top nav of the AWS Console, choose Services, then All AWS Services, then API Gateway.
  2. Click the Get Started button.
  3. Click the OK button.
  4. Select the New API radio button.
  5. For API name, specify the same name of your project as you did for the S3 bucket.
  6. Click the Create API button.
  7. Click the Actions… button, and then choose Create Method.
  8. From the select box, choose POST and then click the check button.
  9. For Integration type, choose the Lambda Function radio button.
  10. For Lambda Region, choose the region for your function, which should be us-east-1.
  11. For Lambda Function, specify the first character name of your function (the same name as your project), and then choose the matching function name.
  12. Click the Save button.
  13. Click OK.
  14. Click Integration Request.
  15. Under Body Mapping Templates, click the Add mapping template button,
  16. For Content-Type, specify application/x-www-form-urlencoded.
  17. Click the check button.
  18. Click the Yes, secure this integration button.
  19. For application/x-www-form-urlencoded, specify $input.json('$').
  20. Click the Save button.
  21. Click ← Method Execution button to go back.
  22. Click Integration Response.
  23. Under Body Mapping Templates, click the – button next to application/json.
  24. Click the Delete button.
  25. click the Add mapping template button.
  26. For Content-Type, specify application/xml.
  27. Click the check button.
  28. For application/xml, specify xml version="1.0" encoding="UTF-8"?><Response></Response>.
  29. Click the Save button.
  30. Click ← Method Execution button to go back.
  31. Click Method Response.
  32. Under Response Models for 200, click the Add response model button,
  33. For Content-Type, specify application/xml.
  34. For Model, choose Empty.
  35. Click the check button.
  36. Click the X button next to application/json to delete that content type.
  37. Click the Actions… button and choose Deploy API.
  38. For Deployment stage, choose [New Stage].
  39. For Stage name, specify prod.
  40. Click the Deploy button.
  41. Take note of the URL given at the top in Invoke URL. This is the URL you’ll use for your Twilio webhook.
  42. Your endpoint is ready.

Buy a Twilio number

  1. Open the Twilio Console for phone numbers.
  2. Click the + button.
  3. For COUNTRY, choose United States.
  4. For CAPABILITIES, select MMS.
  5. Click the Search button.
  6. Choose the number you want, and click the Buy button.
  7. Click the Buy This Number button.
  8. Click the Setup number button.

Point your Twilio number at API Gateway

  1. Under Messaging, for A MESSAGE COMES IN,
  2. Choose Webhook.
  3. Specify the invoke URL from your API Gateway endpoint. It should look like https://**********.execute-api.us-east-1.amazonaws.com/prod.
  4. Choose HTTP POST
  5. Click the Save button.
  6. Your phone number is ready.

Test your setup

Once you’ve completed the setup, take out your phone and send a photo to your Twilio phone number. Your frame should be updated with the new photo within about 15 seconds, but if it isn’t:

  1. Check the programmable SMS logs in your Twilio console to see that an appropriate response was returned from API Gateway.
  2. Check the AWS CloudWatch logs for your Lambda function to see if the function terminated successfully.

Also, to check whether your collage has been updated without checking your Electric Objects display directly, just access the collage image from your S3 bucket, in the AWS S3 console.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.