Building a Land Acknowledgement Text Line with Node.js, Twilio, and Puppeteer

Time to read: 5 minutes

May 21, 2020
Written by
Sam Agnew
Twilion
Reviewed by
Tilde Thurium
Twilion

Copy of Generic Blog Header 3.png

In the age of impending climate catastrophe, it is important now more than ever to respect the rights and sovereignty of Native people worldwide. Recognizing the traditional stewards of the land by making land acknowledgements at events or gatherings is a vital first step towards inserting an awareness of Indigenous presence and land rights into everyday life.

This kind of thing has been becoming a bit more commonplace at developer conferences such as JSConf US and PyCascades.

PyCascades Land Acknowledgment

Using Native Land Digital, an Indigenous-led, in-progress project dedicated to helping people learn more about their local traditional Indigenous territories, I created a text messaging bot for people who are on the go to quickly find out which Native territory a given address exists on.

Try it by texting an address to +1 (331) 244-LAND (+1 331 244 5263).

By texting "101 Spear Street" for example, we can see that the Twilio office in San Francisco exists on Ramaytush Ohlone land.

Example of the working app

Let's walk through the technical aspects of how to build a tool like this. We'll be using Node.js to interact with Twilio Programmable SMS for handling incoming text messages, as well as Puppeteer to retrieve data from Native Land Digital via headless browser scripting. Although an API does exist, it only takes location data in the form of geo-coordinates rather than a more user-friendly address. This is why our code will type an address on the website that was received via SMS to respond with the corresponding Native land.

Setting up dependencies

Before moving on, make sure you have an up to date version of Node.js and npm installed.

Navigate to the directory where you want this code to live and run the following command in your terminal to create a package for this project:

npm init --yes

The --yes argument runs through all of the prompts that you would otherwise have to fill out or skip. Now that we have a package.json file for our project, run the following command in your terminal to install the dependencies:

npm install puppeteer@2.1.1
npm install twilio@3.39.5
npm install body-parser@1.19.0
npm install express@4.17.1

When you install Puppeteer, it downloads a recent version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win) that is guaranteed to work with the API. This download might take a while to install, so hang tight.

Getting results from the Native Land tool using Puppeteer

Using Puppeteer, we can write scripts to interact with web pages programmatically. In a previous tutorial, I already wrote some code to interact with this same website. We'll build off of this in a second, but first let's take a look at the steps we need to take in human terms.

When visiting the Native Land website, we want to dismiss the disclaimer to access the map:

Disclaimer

Then we want to click on the search box and type a location:

Search Box

And finally, we want to click on the first suggestion that pops up, and grab the text that appears in the results section that lists the relevant Native territories, Lenape in this case:

Results

The code in my previously mentioned Puppeteer tutorial walks through how to do all of that in more detail, so follow that if you want to dig deeper. We're going to build off of that code now.

Create a file called index.js and add the following to it:

const puppeteer = require('puppeteer');

async function getLandText(location) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const MODAL_BUTTON_SELECTOR = '.modal-footer > button';
  const SEARCH_SELECTOR = 'input[placeholder=Search]';
  const LOCATION_SELECTOR = 'li.active';
  const RESULTS_SELECTOR = '.results-tab';

  await page.goto('https://native-land.ca/');
  await page.click(MODAL_BUTTON_SELECTOR);
  await page.waitFor(2000);

  await page.click(SEARCH_SELECTOR);
  await page.keyboard.type(location);
  await page.waitForSelector(LOCATION_SELECTOR);

  await page.click(LOCATION_SELECTOR);

  // If certain elements on the page haven't finished loading before we do the search,
  // then the location data won't actually display.
  // This loop retries the search until a result is given.
  while (!await page.$(`${RESULTS_SELECTOR} > p`)) {
    await page.waitFor(500);
    await page.click(SEARCH_SELECTOR);
    await page.keyboard.type(location);
    await page.waitForSelector(LOCATION_SELECTOR);

    await page.click(LOCATION_SELECTOR);
  }

  const addressElement = await page.$(SEARCH_SELECTOR);
  const address = await addressElement.evaluate(element => element.value);
  const resultElement = await page.$(RESULTS_SELECTOR);
  const resultText = await resultElement.evaluate(element => element.innerText);

  const text = `${address} is on the land of: ${resultText.substring(resultText.indexOf('\n'))}`;

  await browser.close();

  return text;
}

This code uses Puppeteer to do the things we just described. In practice, sometimes the page doesn't load as quickly as the code itself runs, which can result in errors. For this purpose, I included some logic to wait for a bit and to continuously retry the location search if the results aren't loading yet. If you're having any issues, you can tweak the number of milliseconds passed to the page.waitFor method to work better with your internet connection.

To test this code, temporarily add the following to the end of index.js and then run node index.js in the terminal, using whatever location you wish:

(async () => {
  const text = await getLandText('101 Spear Street, San Francisco');
  console.log(text);
})()

You should see something like this logged to the console:

Terminal output

Setting up a Twilio phone number

Before being able to receive text messages, you’ll need a Twilio phone number. You can buy a phone number here (it’s free for you to personally use while testing your code, but if you need anything beyond that just shoot me an email).

The Express app will need to be visible from the internet in order for Twilio to send requests to it. We will use ngrok for this, which you’ll need to install if you don’t have it already. Once installed, run the following command:

ngrok http 3000

If you’ve just installed ngrok and that previous command didn’t work, you might have to run it like ./ngrok http 3000 from the directory that the ngrok executable is in.

ngrok display

We now have a publicly accessible URL to the Express app. Configure your phone number in your Twilio Console as seen in this image by adding your ngrok URL with "/sms" appended to it to the "Messaging" section:

Twilio phone number configuration

You are now ready to receive text messages to your number. Now we need to write some code to handle these messages!

Handling incoming text messages with Express

Let's build an Express app that uses the function we wrote earlier. Replace the code in index.js with this:

const http = require('http');
const express = require('express');
const { urlencoded } = require('body-parser');
const MessagingResponse = require('twilio').twiml.MessagingResponse;
const puppeteer = require('puppeteer');

const app = express();
app.use(urlencoded({ extended: false }));

async function getLandText(location) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const MODAL_BUTTON_SELECTOR = '.modal-footer > button';
  const SEARCH_SELECTOR = 'input[placeholder=Search]';
  const LOCATION_SELECTOR = 'li.active';
  const RESULTS_SELECTOR = '.results-tab';

  await page.goto('https://native-land.ca/');
  await page.click(MODAL_BUTTON_SELECTOR);
  await page.waitFor(2000);

  await page.click(SEARCH_SELECTOR);
  await page.keyboard.type(location);
  await page.waitForSelector(LOCATION_SELECTOR);

  await page.click(LOCATION_SELECTOR);

  // If certain elements on the page haven't finished loading before we do the search,
  // then the location data won't actually display.
  // This loop retries the search until a result is given.
  while (!await page.$(`${RESULTS_SELECTOR} > p`)) {
    await page.waitFor(500);
    await page.click(SEARCH_SELECTOR);
    await page.keyboard.type(location);
    await page.waitForSelector(LOCATION_SELECTOR);

    await page.click(LOCATION_SELECTOR);
  }

  const addressElement = await page.$(SEARCH_SELECTOR);
  const address = await addressElement.evaluate(element => element.value);
  const resultElement = await page.$(RESULTS_SELECTOR);
  const resultText = await resultElement.evaluate(element => element.innerText);

  const text = `${address} is on the land of: ${resultText.substring(resultText.indexOf('\n'))}`;

  await browser.close();

  return text;
}

app.post('/sms', async (req, res) => {
  const twiml = new MessagingResponse();
  const message = req.body.Body;
  const landText = await getLandText(message);

  twiml.message(landText);
  res.writeHead(200, {'Content-Type': 'text/xml'});
  res.end(twiml.toString());
});

http.createServer(app).listen(3000, () => {
  console.log('Express server listening on port 3000');
});

This sets up an Express app, defines a route on the app to handle incoming SMS messages, and then has it listen on port 3000 for incoming requests. In the /sms route which handles requests representing text messages to our Twilio number, the body-parser middleware is used to access data in the request body to grab the text of the message. The text is passed to the function we wrote earlier to create a response in the form of TwiML, which will tell Twilio to send a text message to whoever texted this number.

Run it with the following command:

node index.js

Now that you have your Express app running, and your Twilio phone number configured with your ngrok URL, shoot a text message to your Twilio number and see if it works!

That's cool and all, but what else can I do?

Building technology can be fun, and hopefully this was an enjoyable project for you to dig into. Land acknowledgements can be a good starting point, but as Native Land Digital's informational page on Territory Acknowledgements says, "these acknowledgements can easily be a token gesture rather than a meaningful practice." Here are some things you can do to make sure there are more than just words behind your land acknowledgements:

Event organizers (when in-person events are a thing again) can also reach out to and coordinate with local Indigenous groups when organizing their event. Not just on coming up with a proper land acknowledgement, but also for things like:

  • Offering free attendance to interested Indigenous community members.
  • Reserving speaking slots for Indigenous people.
  • Donating a portion of sponsorship or ticket revenue (if any exists) to support local Indigenous projects.

It's important to make an effort to learn about the history and traditions of the people who are native to wherever you live. Although it's still a work in progress, typing your address into the map on native-land.ca is a great place to begin your research. On top of that, if you're a technologist you can also check out groups like Natives in Tech or listen to the Wampum.codes podcast to learn about Indigenous people who do cool things with technology.

For Native Land Digital, an API is also available if you want to do more with this data. I would recommend using that for further projects. If you build anything with this tool, I would love to see it! I can be found online in the following places:

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.