Build a custom Twilio alerting system using Express.js, json-rules-engine, and MongoDB

Build a custom Twilio alerting system using Express.js, json-rules-engine, and MongoDB

Ahoy builders! In the world of technology, fixing errors quickly is key. Alerting is crucial because it helps us stay proactive rather than reactive. When something goes wrong in our systems or applications, it is often time-sensitive. Alerts notify us when there is an issue, allowing us to take swift action to mitigate any potential damage or downtime.

Twilio’s Console Debugger allows developers to configure an optional webhook to receive data about errors and warnings as they happen. This makes it straightforward for developers to react to any problems with their applications, promptly. While Twilio’s Console Debugger excels in error detection, its notification capabilities are further enhanced by Twilio Alarms, extending the alerting functionality to email notifications. You can set alerts for any warning or error, or for specific errors while defining a threshold value.

While Twilio Alarms offer robust notification capabilities, they may not address every specialized requirement. In some cases, you may find the need to create a custom alerting system tailored to specific requirements. For instance, you might want to set an alert for a group of error codes or for a particular Twilio resource such as a Twilio Function or a Studio flow. Additionally, you might require a single alert to monitor multiple accounts, including subaccounts.

In this tutorial, you will be building a fully customizable alerting system using Express.js, json-rules-engine, MongoDB, the Twilio Debug Events Webhook, and Twilio SendGrid to send email.

Prerequisites

To proceed with this tutorial, you will need the following:

• Node.js v16+ on your development machine.
• Have a Twilio account. If you don’t have one yet, you can register for a trial account here.
• A SendGrid account.
• A MongoDB account. You will be using MongoDB Atlas in this tutorial.
• Install ngrok and make sure it’s authenticated.
  
  • You can install ngrok from here. You can refer to this blog post to create a new account.

Set up your development environment

Create the project

Before building the application, you need to set up your development environment. Open up your terminal and execute the following commands:

This command creates an express-alert directory and navigates into it. The `npm init` command will initialize a project and create a package.json file. The `-y` flag will choose the default settings while creating the project.

Install the dependencies

You will be using the following packages in your application.

• express - Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.
• twilio - A package that allows you to interact with the Twilio API.
• dotenv - A zero-dependency module that loads environment variables from a .env file into process.env.
• json-rules-engine - A powerful, lightweight rules engine. Rules are composed of simple json structures, making them human readable and easy to persist.
• @sendgrid/mail - This library allows you to quickly and easily use the Twilio SendGrid Web API v3 via Node.js.
• mongodb - The official MongoDB driver for Node.js.

Install all the packages using the following command:

Once the packages are installed, you will see an output similar to the following:

Set up environment variables

You can use environment variables to store sensitive data required later in your application. Create an .env file in your project directory and paste the snippet given below:

You can find your Main Account SID from Twilio Console.

Configure SendGrid

You will be using Twilio SendGrid to send alerts via email. Follow the steps below to get started with Twilio SendGrid.

Sign up for SendGrid and create an API key

The first thing you need to do is create a SendGrid account. You can choose the free tier for this tutorial. After setting up your account, generate an API key with “Full Access”, as shown in the screenshot. You're free to choose any name for it, but ensure you save it before proceeding!

Now, update your .env file with the SendGrid API Key as shown below:

Verify your Sender Identity

To ensure the sender’s reputation, Twilio SendGrid requires you to verify the identity of your “From” email address by completing domain authentication.

You can verify one or more Sender Identities using either Domain Authentication or Single Sender Verification. You can go to the Sender Authentication page in the SendGrid Console and click on Verify a Single Sender. Once complete, add the necessary details (you can refer to this post if you have any questions) and click on Create.

Once created, you will receive an email to the sender address. Make sure to open the link contained in the email and verify the sender.

To learn more about Sender Verification, you can refer to this documentation.

After verifying the sender, update your environment variables to include both From and To email addresses as shown below:

Create a SendGrid dynamic template

In this tutorial, you will be using a SendGrid dynamic template to customize your alerts.

Open the Dynamic Templates page and click on the Create a Dynamic Template button. After creating the template, add a Version to it. Select Code Editor to design your email. Copy and paste the html content given below.

Configure MongoDB

You will be using MongoDB Atlas to store the alert count and timestamps. MongoDB Atlas is a fully-managed database service offered by MongoDB, providing the capability to store data in key-value pairs. Additionally, it offers a free plan featuring limited storage. Given our use case of storing only two fields per record (alert count and timestamps), MongoDB Atlas is an ideal solution.

Follow the steps below to get started with MongoDB Atlas.

Sign up for MongoDB Atlas and create a Cluster

Create a MongoDB Atlas account using one of the available sign in options. When you first create an account in MongoDB Atlas, you will see the following screen.

For this tutorial, we will opt for the free Shared cluster, but feel free to adjust its configuration to suit your requirements. Click on Create to continue.

Once the Cluster is created, you will be asked to complete the security quickstart. In this tutorial, you will be using username and password authentication. So, provide a username and password, as shown below, and click on Create User.

Create a Database and Collection

Now that your cluster is prepared, let’s set up a database to store our collection.

Click on the Add Data option as depicted below to proceed. You will then be directed to a page where you can create a new database and collection.

You will be creating the Database on Atlas, so click on START as shown below.

Copy the connection details and update .env file

Click on the CONNECT button on the overview page to get connection details (as shown below).

Replace <password> with your actual password.

Build your Application

Create an Express Server

You will be using Express.js to build this application. An Express server is a web application framework for Node.js, designed to build web applications and APIs. It simplifies the process of creating robust and scalable server-side applications by providing a set of powerful features such as routing, middleware, and other features you will need in a server.

Let’s set up a basic express server. We will use it to handle webhooks from Twilio, which include debugging information we will use to form our alerts.  

In your project directory (express-alert), create a file called app.js and paste the code snippet given below:

Here you are using ES Modules, so make sure to add "type": "module" in your package.json file. You can refer to this document if you want to learn more.

As you can see from the first line, you import express using the import keyword and an instance of the Express application is created by calling the express() function. Finally, you call the listen() method on your Express application, passing in the port number and a callback function that will be executed once the server starts.

The Twilio webhook passes the parameters in url-encoded format, so you need to use the express.urlencoded({ extended: true }) middleware.

The express.urlencoded() function is a built-in middleware function in Express. It parses incoming requests with URL-encoded payloads and is based on a body parser. This middleware is available in Express v4.16.0 onwards.

Create the rule engine

Here you will be creating a Rule Engine. You will use json-rule-engine to filter the incoming webhook payload and send an email alert only for specific events.

json-rules-engine is a powerful, lightweight rules engine. Rules are composed of simple json structures, making them human readable and easy to persist.

Create a file called ruleEngine.js and paste the code code snippet given below.

In the above snippet, you'll develop a Rule Engine to assess three rules: detecting errors thrown by any function across your sub accounts, identifying errors thrown by a specific function SID in the main account, and recognizing errors thrown by any Studio flow in your main account.  

To implement this, you need to import Engine from json-rules-engine. Twilio passes data to your webhook in the form of application/x-www-form-urlencoded. Within that request body, the Payload property is a JSON object that you will need to decode. You can find the full list of parameters here.

The rulejson1 contains a rule to match errors thrown by any function across all your sub accounts.

The rulejson2 contains a rule to match errors thrown by a specific function SID in the main account.

The rulejson3 contains a rule to match errors thrown by any studio flow in your main account.

The variables STUDIO_ERROR_CODES_BEGIN and STUDIO_ERROR_CODES_END contain the beginning and ending values of Studio related errors. Similarly, FUNCTION_ERROR_CODES_BEGIN and FUNCTION_ERROR_CODES_END contain the beginning and ending values of Functions related errors. You can find the full list of error codes here.

The variable thresholdRule sets a threshold for each rule to ensure the triggering of alarms. The threshold values are set in milliseconds.

Once the JSON rules are created, you need to create an engine and pass replaceFactsInEventParams: true to include the webhook data in the event parameters. Add all the three rules to the Engine object using the addRule method. Now, export the engine and thresholdRule using export { engine, thresholdRule } to use it in other files.

Create a route for incoming webhook

Now you will create an endpoint in your express application to handle incoming webhook requests. Paste the following code snippet in your app.js file:

Here the alertMiddleware is the middleware function you will be creating shortly, and it is used to parse incoming webhook data and apply the logic to send email alerts.

Implement the middleware logic

Now, you will be implementing the middleware to handle the webhook requests. Create a new file called alertMiddleware.js in the root directory and paste the code snippet given below.

The code above defines an `alertMiddleware` function, which serves as middleware for processing incoming webhook data. This function extracts the JSON payload from the incoming request, parses it, and then calls the validateRules function to evaluate the defined rules.

The validateRules function will first retrieve all the threshold data from the MongoDB collection. Then it adds the webhook data as a fact to the rule engine. Then it runs the rule engine and iterates over the events triggered by the rules. For each event, it retrieves the relevant data from the MongoDB collection array docArray based on the event type. Then, it calculates the time difference between the current event and the last occurrence of the event in the database.

Depending on the calculated time difference and the configured threshold for the event type, it updates the database and potentially triggers an email alert if the threshold condition is met.

Implement the SendGrid logic

Now that we have completed the middleware, it is time to implement the sendEmailAlert function we have used in the middleware. Create a file sendEmail.js and paste code snippet given below:

Run your Express Server

Let's kick off your server so you can handle the webhook requests. Navigate to your project directory (twilio-alerts) and run this command.

It will start an express server at port 3000 and you can access the server using the URL

http://localhost:3000. You should see a “Hello World!” message when you open this URL in the browser.

The Express server can't be reached online because it's local to your machine. Twilio needs to send web requests to this server, so during development, you'll have to make the local server available to the Internet. To do this, we’ll use ngrok.

Open a second terminal window and run the following command:

When ngrok is active, you can reach the application from anywhere globally using the temporary forwarding URL displayed in the command output. Any web requests sent to the ngrok URL will be directed to the Express application by ngrok.

Configuring the Twilio webhook

Once you've set up the webhook, the next step is to configure the debugger webhook URL in the Twilio console. This ensures that you receive webhook requests whenever an alert is logged. Navigate to the Twilio Console and paste the ngrok forwarding URL with https in it into Webhook URL (as shown below).

Don't forget to check the box for Include sub account errors and warnings to also capture errors and warnings from subaccounts.

Test your Application

It’s time to test your application. Your express server will send an email notification whenever the error logs match the rules we have set using the rule engine.

To test your application, you can create a simple Studio flow with a Make HTTP Request Widget which tries to make a request to an invalid URL (for example, you can set the URL to https://urlnotfound/). When this flow is executed, it should throw an error, and you should receive an email alert as shown below

Note that the webhook payload displayed in the email depends on the error thrown since the webhook property of the payload is optional. You can always customize the look and feel of your email for your use case using dynamic templates. The More Details button will have a link to your error log in the debugger console.

Conclusion

By using Twilio Debug event webhooks, a JSON-rule engine, MongoDB, Express.js, and SendGrid you've just created a customizable alerting system over email. This system ensures peace of mind, as you will get quick notifications if something important goes wrong in your application.

What's great is that this solution can be changed in a straightforward manner. Right now, it uses the Debug Event Webhook, but you can add other webhook sources too. This means you can make the system better and add rules based on different webhook formats.

With this strong alerting system, you can stay on top of things and make sure your app runs smoothly, fixing problems before they become big issues.

Vikhyath is a Technical Support Engineer at Twilio, dedicated to assisting customers in achieving success with Twilio. He is very passionate about Technology and enjoys sharing his knowledge with others. He can be reached at vs [at] twilio.com