Build a Hobby Recommender with Twilio Functions and the WikiMedia API

Time to read: 5 minutes

July 25, 2022
Written by
Dainyl Cua
Twilion
Reviewed by
Hayden Powers
Twilion
Johnny Nguyen
Twilion

Build a Hobby Recommender with Twilio Functions and the WikiMedia API

In honor of National Anti-Boredom Month, I decided to build a bot that gives me suggestions for hobbies to pick up in the near future. Want to get hobby suggestions for yourself? Try texting any message to the number +1 (717) 942 - 9721, and see what you get!

In this tutorial, you’ll see how I built the bot (aptly named HobbyBot) and learn how to build it yourself with Twilio Runtime, specifically Twilio Functions, and the WikiMedia API.

Prerequisites

To follow along to this tutorial, you will need:

  • A free or paid Twilio account. If you are new to Twilio, click here to sign up for a free Twilio account and get a $10 credit when you upgrade!
  • A Twilio phone number
  • A personal phone number

Setting up your developer environment

For this tutorial, you will only need to work with Twilio Functions. The first step in doing so is to head to the Services page located under the Functions and Assets tab on the left side of the screen in your Twilio Console. Next, click on the blue button in the top half of the screen labeled Create Service. Enter a name for your Service (for this tutorial, I named the Service hobby-bot), and click the blue button in the bottom left of the popup window labeled Next. You should be directed to a code editor, and it should look similar to this:

Empty Twilio Functions editor page

Click the blue button in the top left labeled Add +, click on Add Function in the dropdown menu, then rename the new function path if you wish (for this tutorial, I named the function path /main).

Next, click on Dependencies in the bottom left of the page, below the Settings header. Enter node-fetch in the Module text box and enter 2.x in the Version text box. Then, click the button to the right labeled Add. This imports the node-fetch npm module to use in your application and allows you to utilize the Fetch API to request and pull in data from the WikiMedia API.

If you’ve worked with APIs before, you may notice that you do not need to set up any environment variables to store your API key credentials.

As opposed to most APIs, the WikiMedia API does not require you to sign up for an API key to gain access to the API endpoints. Instead, please be mindful of request etiquette when developing an application that uses their API!

Functions dependencies configuration page. Node.js version set to v14. node-fetch version set to 2.x

With your environment fully set up, you can now start writing code!

Building your bot

Click on the /main tab near the top of your page to switch back to your code editor. Replace the default code with the following:

const fetch = require('node-fetch');

exports.handler = async function(context, event, callback) {
  // Initialize MessagingResponse
  let twiml = new Twilio.twiml.MessagingResponse();
  let message = twiml.message();

  return callback(null, twiml);
};

These lines of code initialize the node-fetch module that you imported earlier, as well as the twiml and message variables that you will be working with. These two variables are objects with parameters responsible for storing data used when sending text messages. The callback() function at the end of the function will be responsible for sending your message once the code is complete.

Next, insert the following code below line 6 where you initialized the message variable:

  // Attempt to fetch data, then parse it
  let unformattedHobbies = {};
  try {
    const res = await fetch("https://en.wikipedia.org/w/api.php?action=parse&page=List_of_hobbies&format=json&prop=wikitext");
    unformattedHobbies = await res.json();
  } catch(e) {
    console.log("An error has occurred when fetching all hobbies, ", e);
  }

  const hobbyRegEx = /((?<!\=\[\[)(?<=(\[\[))).+?(?=(\]|\|))/g;
  const fullHobbies = unformattedHobbies.parse.wikitext["*"].match(hobbyRegEx);
  const hobbies = [...fullHobbies.slice(3,244), ...fullHobbies.slice(245,673)];

This code utilizes a try catch block to run the data fetching code and catch and parse any errors that occur. The API endpoint URL in fetch() utilizes the parse action to get all the text on the List of hobbies page, formatted in wikitext. If the data is successfully gathered, it is then parsed into JSON and stored in unformattedHobbies.

The data is then parsed using a regular expression and the .match() function, transforming it into an array. Invalid entries in the array are then removed and then a sanitized array filled with hobby names is stored in the hobbies variable.

The regular expression stored in hobbyRegEx can be difficult to parse, and understanding how it works is out of the scope for this tutorial. If you want to play around with regular expressions, try a site like regexr.com and look at the documentation above!

Next, add the following code directly below the one you just added:

// Select a hobby from the list and format the name for another fetch
  const randNum = Math.floor(Math.random() * (hobbies.length));
  const hobby = hobbies[randNum]
  let formattedHobbyName = hobby.replace(/\s+/g, "_");

  let unformattedHobby = {};
  try {
    const res = await fetch(`https://en.wikipedia.org/w/api.php?action=parse&page=${formattedHobbyName}&format=json&prop=wikitext`);
    unformattedHobby = await res.json();
  } catch(e) {
    console.log("An error has occurred when fetching single hobby, ", e);
  }

  // Check if there was a redirect, reset formattedHobbyName if redirected
  const redirectRegEx = /(#REDIRECT)/g;
  const isRedirect = unformattedHobby.parse.wikitext["*"].match(redirectRegEx);
  if(isRedirect) {
    unformattedHobby = isRedirect.replace(/\s+/g, "_");
  }

This code generates a random index inside the hobbies array then grabs the hobby name at that index, storing it within the hobby variable. The hobby string is then formatted using a regular expression that replaces all whitespaces with underscores, and stored in the formattedHobbyName variable. Similar to the previous code, another fetch() is performed that uses the parse action. This time, however, you will be pulling text from the Wikipedia page of the hobby you randomly selected and storing it within the unformattedHobby variable.

The code then checks if a redirect occurs when visiting the page—if the redirect is not handled properly, an improper description of the hobby will be sent to the reader. If there is a redirect, then a regular expression and the .match() function will be used to gather the name of the redirect destination page, then the new name is assigned to formattedHobbyName. If not, the code proceeds as normal.

Next, copy and paste the following lines of code below the last ones you added:

// Get the description of the page
  let description = "";
  let thumbnailUrl = "";
  try {
    const res = await fetch(`https://en.wikipedia.org/w/api.php?action=query&titles=${formattedHobbyName}&format=json&prop=pageimages|extracts&pithumbsize=1000&exintro=1&explaintext=1`);
    const data = await res.json();
    const page = Object.values(data.query.pages)[0]
    description = page.extract.replace(/\\n/g);
    if(description.length > 1400) {
      // To circumvent the 1600 character limit, slice if the description is too long.
      description = description.slice(0,1400) + "...";
    }
    thumbnailUrl = page.thumbnail.source;
  } catch(e) {
    console.log("An error has occurred when fetching description and thumbnail, ", e);
  }

These lines of code perform the final fetch you will use for your bot. Instead of utilizing the parse action like you did in the previous fetches, the query action is used instead. With the query action, it’s easier to get the page thumbnail URL (if it exists) and the description of the Wikipedia page. Once you’ve successfully retrieved the data, you can extract the description (and trim it in case it goes above the 1600 character limit for messaging) and get the page thumbnail URL.

Finally, add the last bits of code below the try catch block:

// Spice up the greeting!
  const greetings = [
    "Have you considered trying",
    "Hey! Why don't you look into",
    "Maybe try out",
    "What do you think about picking up",
    "Here's a hobby you can try out",
    "Maybe you can learn",
    "If it's not too hard, why not learn",
    "I think you can try",
    "Here's my suggestion",
    "Have you heard of",
    "Take a look at",
  ];
  const greetingNumber = Math.floor(Math.random() * greetings.length);
  const greeting = greetings[greetingNumber];

  // Send the message 
  message.body(`${greeting}:\n\n${hobby.toUpperCase()}\n\n${description}\n\n\nLink: https://en.wikipedia.org/wiki/${formattedHobbyName}`);
  if(thumbnailUrl) {
    message.media(thumbnailUrl);
  }

This final code first initializes an array of greetings that the bot greets the texter with to add some personality in its responses. A random index inside the greetings array is generated then the greeting at that index is chosen and stored in the greeting variable. Finally, the message object’s body parameter is set to a string that uses the greeting, description, and formattedHobbyName variables. If thumbnailUrl is not an empty string (indicating a thumbnail URL was found), then the message object’s media parameter is set as well.

Your code is now complete! The code in its entirety should look like the following:

const fetch = require('node-fetch');

exports.handler = async function(context, event, callback) {
  // Initialize MessagingResponse
  let twiml = new Twilio.twiml.MessagingResponse();
  let message = twiml.message();

  // Attempt to fetch data, then parse it
  let unformattedHobbies = {};
  try {
    const res = await fetch("https://en.wikipedia.org/w/api.php?action=parse&page=List_of_hobbies&format=json&prop=wikitext");
    unformattedHobbies = await res.json();
  } catch(e) {
    console.log("An error has occurred when fetching all hobbies, ", e);
  }

  const hobbyRegEx = /((?<!\=\[\[)(?<=(\[\[))).+?(?=(\]|\|))/g;
  const fullHobbies = unformattedHobbies.parse.wikitext["*"].match(hobbyRegEx);
  const hobbies = [...fullHobbies.slice(3,244), ...fullHobbies.slice(245,673)];
  // Select a hobby from the list and format the name for another fetch
  const randNum = Math.floor(Math.random() * (hobbies.length));
  const hobby = hobbies[randNum]
  let formattedHobbyName = hobby.replace(/\s+/g, "_");

  let unformattedHobby = {};
  try {
    const res = await fetch(`https://en.wikipedia.org/w/api.php?action=parse&page=${formattedHobbyName}&format=json&prop=wikitext`);
    unformattedHobby = await res.json();
  } catch(e) {
    console.log("An error has occurred when fetching single hobby, ", e);
  }

  // Check if there was a redirect, reset formattedHobbyName if redirected
  const redirectRegEx = /(#REDIRECT)/g;
  const isRedirect = unformattedHobby.parse.wikitext["*"].match(redirectRegEx);
  if(isRedirect) {
    unformattedHobby = isRedirect.replace(/\s+/g, "_");
  }

  // Get the description of the page
  let description = "";
  let thumbnailUrl = "";
  try {
    const res = await fetch(`https://en.wikipedia.org/w/api.php?action=query&titles=${formattedHobbyName}&format=json&prop=pageimages|extracts&pithumbsize=1000&exintro=1&explaintext=1`);
    const data = await res.json();
    const page = Object.values(data.query.pages)[0]
    description = page.extract.replace(/\\n/g);
    if(description.length > 1400) {
      // To circumvent the 1600 character limit, slice if the description is too long.
      description = description.slice(0,1400) + "...";
    }
    thumbnailUrl = page.thumbnail.source;
  } catch(e) {
    console.log("An error has occurred when fetching description and thumbnail, ", e);
  }

  // Spice up the greeting!
  const greetings = [
    "Have you considered trying",
    "Hey! Why don't you look into",
    "Maybe try out",
    "What do you think about picking up",
    "Here's a hobby you can try out",
    "Maybe you can learn",
    "If it's not too hard, why not learn",
    "I think you can try",
    "Here's my suggestion",
    "Have you heard of",
    "Take a look at",
  ];
  const greetingNumber = Math.floor(Math.random() * greetings.length);
  const greeting = greetings[greetingNumber];

  // Send the message 
  message.body(`${greeting}:\n\n${hobby.toUpperCase()}\n\n${description}\n\n\nLink: https://en.wikipedia.org/wiki/${formattedHobbyName}`);
  if(thumbnailUrl) {
    message.media(thumbnailUrl);
  }
  
  return callback(null, twiml);
};

Finally, hit the blue button labeled Save below your code editor and then hit the blue button labeled Deploy All on the bottom left of the page. All that’s left to do is connect your Service to your Twilio phone number.

Adding the function to your Twilio phone number

Navigate to the Active Numbers page located in the Manage dropdown menu under the Phone Numbers tab in the left side of your Twilio Console. Click on your active number, then scroll down to the Messaging section.

Under the heading that says A MESSAGE COMES IN, click the dropdown menu and select Function. Next, select your Service in the dropdown menu under the SERVICE header, then select ui in the dropdown menu under the ENVIRONMENT header, and finally the function path you named earlier under the FUNCTION PATH header.

A Twilio phone number is configured to send text messages to a Twilio function service named hobby-bot

After that, hit the blue button labeled Save at the bottom of the page. You should now be able to text your Twilio phone number and get a recommendation!

Response from recommendation bot, received Judo as a recommendation

Conclusion

Hopefully this tutorial helped you understand how I built HobbyBot! With Twilio Runtime, you don’t have to worry about setting up a server that will host your application forever—you only need to build the code and let Twilio handle everything else. Maybe you could integrate it with your own products to kill some time during National Anti-Boredom Month?

I can’t wait to see what you build next!

Dainyl Cua is a Developer Voices Intern on Twilio’s Developer Network. They would love to talk at any time and help you out. They can be reached through email via dcua[at]twilio.com or through LinkedIn.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.