Build an Incident Alert Bot with Twilio WhatsApp, Winston and Node.js

Time to read: 10 minutes

November 02, 2023
Written by
Desmond Obisi
Contributor
Opinions expressed by Twilio contributors are their own
Reviewed by
Dhruv Patel
Twilion

Looking for important logs in the pools of log files and data can be a pain at times during development, testing or debugging processes.

If we can have a tool that gives a real-time report of critical, error and warning reports about the activities in our APIs, it will really make triaging and bug fixing a lesser issue for developers. Imagine a scenario when you get an alert on WhatsApp (personal or group) of incidents happening in your API as they happen, developers can readily remedy costly bugs in no time and maintain a good customer experience.

Through this tutorial, you will learn how to integrate Twilio's WhatsApp API and Winston to a Node.js API, making incident/error reporting and troubleshooting as easy as possible.

Prerequisites

Here is a list of what you need to follow along in this tutorial

Setting Up Your Application

To set up your APIs, I attached a link to a codebase that contains the base application used for this tutorial. It contains all the code necessary to start a Node.js server and some already-made endpoints that work once you connect to a MySQL database. This section will work you through how to run the project on your local machine, set up Twilio and any other requirements you need to build your solution

Running the Node.js APIs

To get the project running on our local machine, you can follow the steps below:

Navigate to your terminal and clone the project from the GitHub repository by running the following command:

git clone https://github.com/DesmondSanctity/twilio-log-alert.git

Make sure you are in the APIs-only branch and then run the installation script within the project directory to install the needed packages:

npm install

After the packages have been installed, open up the project directory on your preferred IDE and then create a .env file and add the code below with their values:

PORT=5000
DB_NAME=XXXXX
DB_USERNAME=XXXXX
DB_PASSWORD=XXXXX
DB_HOST=XXXXX
JWT_SECRET=XXXXX
JWT_EXPIRES_IN=XXXXX
TWILIO_AUTH_TOKEN=XXXXX
TWILIO_ACCOUNT_SID=XXXXX
REDIS_URL=XXXXX

To set up MySQL on your local machine, you will download the XAMPP installer for your operating system from their official website and install it on your local machine. After installation, you will get this screen below when you run the application.

XAMPP control panel at start.

Start the Apache and MySQL server by clicking the Start button in the Actions section for Apache and MySQL. When the server is started, you can navigate to PHPMyAdmin with the following link: http://localhost/phpmyadmin/. Or you can click on the Admin button where a web page will open to the XAMPP dashboard where you can access phpMyAdmin from. The phpMyAdmin page is shown below:

An overview of phpMyadmin dashboard

To create a new database, you will click the New button by the left sidebar, add the name you want your database to have and create it.

Creating a database in the phpMyAdmin dashboard

You can choose to set up a user account with a password to access your database or use the default root user with all Admin privileges and connect to your database. Click on the User accounts tab at the top to access the user accounts page. You will see all the users available, their username, password if any, host address and privileges. For this tutorial, you will use the root user with root as its username, localhost as its host, no password and all privileges.

User accounts in the phpMyAdmin dashboard

Then, proceed to use the details in your .env file.

DB_NAME=alertdb
DB_USERNAME=root
DB_PASSWORD=
DB_HOST=localhost

Now our database is ready for connection. Once we establish a connection to it when we run our application, the necessary tables will auto-create and we can start reading or writing to the database.

Running the code below can generate a random 64-bit ASCII string that can be used for encrypting JWT tokens. After you run the code, copy the token as the value for JWT_SECRET in your .env file:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

The JWT_EXPIRES_IN variable in your .env file is used to moderate the time our JWT tokens will expire and users will have to login again to use the APIs. In this tutorial you will use 12h as the value which signifies 12 hours.

You can fetch the REDIS_URL from any Redis instance set up by any provider. In this project, you will use Render’s Redis provisioning. After creating an account on Render, you can click on the New button in the dashboard to set up a Redis server as shown below:

Render dashboard with option to create a Redis server

Enter a name for your Redis instance, choose the free tier and then click the Create Redis button to create the server as shown below:

Redis server creation interface

After creating the server, go to the dashboard, click on the Redis server you created to get the credential you will use to connect to it. Scroll down to the Access Control section and click Add source. Enter 0.0.0.0/0 for the Source and click Save. This allows access to your Redis instance from any server whether you're hosting on your own local environment or a cloud server.


Redis information to use for connection to it

Now scroll up to the Connections section and copy the External Redis URL. Paste this value in your .env file for the REDIS_URL variable.

Alternatively, you can use a local instance of Redis on your machine if you have one set up already.

For Twilio credentials, you can get them from your account dashboard if you already have an account with Twilio or follow the step in the next section to set it up and you will see the keys as shown in the next section.

Remember to add the .env file and any other file you may have that contains secret keys to the .gitignore file to avoid exposing them to the public

Setting up Twilio Account

To set up your Twilio account, sign up for an account and log into your Twilio Console using your account details. From the toolbar's Account menu, select API Keys and Tokens. Take note of your test credentials as shown in the photo below, which include the Account SID and Auth Token. Head over to your .env file and add these values to the TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN variables respectively.

This photo shows the API keys from Twilio console

To complete the setup process, access your Twilio Console dashboard and navigate to the WhatsApp sandbox in your account. This sandbox is designed to enable you to test your application in a development environment without requiring approval from WhatsApp. To access the sandbox, select Messaging from the left-hand menu, followed by Try It Out and then Send A WhatsApp Message. From the sandbox tab, take note of the Twilio phone number and the join code.

A photo showing how to connect to the Whatsapp sandbox from Twilio console

Adding Logging and Alert Functionality with Winston and Twilio

In this section, you will delve into adding logging and alert functionality to your application. You will learn about Winston, a library that aims to decouple parts of the logging process in an API to make it more flexible and extensible. You will learn how to use it in a Node.js API, how to format logs, redact sensitive information and set it as a middleware to cover all your endpoints. You will also learn how to add the Twilio WhatsApp API function to send real-time messages for defined incidents in the API.

Creating Logger Helpers Function

In your codebase, you will install three new packages. Install them by entering the command below on your terminal:

npm install winston winston-daily-rotate-file twilio

This command will install the following packages:

  • Winston: The tool you will use to get logs from every request and response in your APIs
  • Winston Daily Rotate File: This package will help you organize your logs and save them in a file by the day they occur. This way, it is easier to find your logs by the day it occurred.
  • Twilio: This is the Node.js package to connect to Twilio services and will be used to set up real-time WhatsApp messaging.

After a successful installation, you are set to create some helper functions for your logging functionality. These files should be created in the src/utils/log directory. The first one is the sensitiveKeys.js file. In this file, you will make a list of items you do not want to appear in the logs. Sensitive data like user information, payment details, and confidential data as we defined it in our database, services and config files are to be stored here. For our API, the list is small but for larger apps, it should contain as much as you want to redact from the log.

Create a folder within the src/utils folder called logs and within the logs folder create a file named sensitiveKeys.js. Once created, add in the following code:

// Define sensitive keys you want to remove from logs
export const SensitiveKeys = {
  UserId: "userId",
  Password: "password", //this is how the value is stored in the config, service or database.
  NewPassword: "newPassword",
  OldPassword: "oldPassword",
  RepeatPassword: "repeatPassword",
  PhoneNumber: "phoneNumber",
  Token: "token",
  Authorization: "authorization",
};

Another helper function you will create is the constants.js file. This file is where we want to store some information about HTTP methods and HTTP headers and Response messages that stay the same throughout the entire codebase hence the name constants. Create a folder named constants in the src/utils folder and create the constants.js file in the new folder and add the code below:

export const SuccessMessages = {
  CreateSuccess: "Resource created successfully",
  GetSuccess: "Resource retrieved successfully",
  UpdateSuccess: "Resource updated successfully",
  DeleteSuccess: "Resource deleted successfully",
  GenericSuccess: "Operation completed successfully",
  UserRemoveSuccess: "User removed!",
  ProductRemoveSuccess: "Product removed!",
};
export const HTTPHeaders = {
  ResponseTime: "x-response-time",
  ForwardedFor: "x-forwarded-for",
};

export const HTTPMethods = {
  HEAD: "HEAD",
  GET: "GET",
  POST: "POST",
  PATCH: "PATCH",
  PUT: "PUT",
  DELETE: "DELETE",
};

The next helper function is the redactedData.js file. This file will take in the object from sensitiveData.js by their keys or value and replace them with ****** in the request or response JSON body that will be parsed by it. This way those sensitive data are not exposed in the log and in the alert sent to WhatsApp. Create the redactedData.js file inside the src/utils/log folder and add the following code inside the file:

import { SensitiveKeys } from "./sensitiveKeys.js";

const sensitiveKeysList = Object.values(SensitiveKeys)

export const redactLogData = (data) => {
  if (typeof data === 'object' && data !== null) {
    if (Array.isArray(data)) {
      return data.map(item => redactLogData(item));
    }

    const redactedData = {};
    for (const key in data) {
      if (sensitiveKeysList.includes(key)) {
        redactedData[key] = '******'; // replace sensitive data with *
      } else {
        // Recursively redact sensitive keys within nested objects
        redactedData[key] = redactLogData(data[key]);
      }
    }
    return redactedData;
  } else {
    return data;
  }
};

Finally, for the helper functions, you will create indentation.js. This will help to define how we want to handle spacing and indentation in the log file seeing we are dealing with JSON objects most of the time. Create the indentation.js file in the src/util/log folder and add in the following code:

export const LogIndentation = {
  None: 0,
  SM: 2, // Small
  MD: 4, // Medium
  LG: 6, // Large
  XL: 8, // XLarge
  XXL: 10,
  XXXL: 12,
};

Creating the Twilio Function

Next, you will write the functions for the alert messaging using Twilio. For this, you will create a new file named alertFunctions.js in the src/utils/alert directory. These functions will be responsible for activities like formatting the message that will be sent to WhatsApp to a readable format and sending the messages as well. Notice how the code uses the Twilio keys obtained from the setup at the beginning of this tutorial to create an instance of a Twilio client for your use.

Create the alert folder in the src/utils folder and create the alertFunctions.js file within it. In your alertFunctions.js file, add the following lines of code:

import twilio from "twilio";
import { accountSid, authToken } from "../../config/index.js";

// Initialize Twilio client with your credentials
const twilioClient = new twilio(accountSid, authToken);

// A formatted message to send to the user
const formatErrorAlert = async ({
  errorDescription,
  affectedEndpoint,
  startTime,
  duration,
  details,
  alertType,
  method,
}) => {
  return `
    *${alertType == "Critical" ? `⛔ Alert Type ` : `🚫 Alert Type: `}${alertType}*\n
    ⚠️ Error Description: ${errorDescription}\n
    🌐 Affected Endpoint: ${affectedEndpoint}\n
    🔗 HTTP Method: ${method}\n
    🕒 Start Time: ${startTime}\n
    ⌛ Duration: ${duration}\n
    📝 Details: ${JSON.stringify(details)}\n
    `;
};
export const sendWhatsAppAlert = async (messageParams) => {
  const message = await formatErrorAlert(messageParams);
  try {
    await twilioClient.messages.create({
      body: `New Incident Alert:\n ${message}`,
      from: "whatsapp:<your Twilio WhatsApp number>",
      to: "whatsapp:<your own number>",
    });
    console.log(`WhatsApp Alert sent successfully.`);
  } catch (error) {
    console.error(`WhatsApp Alert error: ${error.message}`);
  }
};

The function formatErrorAlert does the formatting of the message and structures it in a readable manner while the sendWhatsAppAlert function takes the formatted message as a parameter and sends it to the number designated to receive the alert. It is worth noting the following parameters:

  • Body - this contains the alert content to be sent.
  • From - The sender, who the message is coming from.
  • To - The recipient, who is receiving the message.

Replace the placeholder numbers with the Twilio WhatsApp phone number and your own personal number where the message will be sent respectively. The Twilio WhatsApp number is shown in your console where you connect to the sandbox.

Creating The Logger Function

Here you will start with setting up the middleware that will be plugged into our APIs to capture the logs. You will create a log instance using Winston, define all the configurations and settings and finally define a transport system for outputting the log. In the src/middlewares directory, create a new file name logger.js and add the code below to it:

import { randomBytes } from "crypto";
import winston from "winston";
import { LogIndentation } from "../utils/log/indentation.js";
import DailyRotateFile from "winston-daily-rotate-file";

const { combine, timestamp, json, printf } = winston.format;
const timestampFormat = "MMM-DD-YYYY HH:mm:ss";

const appVersion = process.env.npm_package_version;
const generateLogId = () => randomBytes(16).toString("hex");

export const httpLogger = winston.createLogger({
  format: combine(
    timestamp({ format: timestampFormat }),
    json(),
    printf(({ timestamp, level, message, ...data }) => {
      const response = {
        level,
        logId: generateLogId(),
        timestamp,
        appInfo: {
          appVersion,
          environment: process.env.NODE_ENV,
          proccessId: process.pid,
        },
        message,
        data,
      };

      // indenting logs for better readbility
      return JSON.stringify(response, null, LogIndentation.MD);
    })
  ),
  transports: [
    // log to console
    new winston.transports.Console({
      // if set to true, logs will not appear
      silent: process.env.NODE_ENV === "test_env", // true/false
    }),
    // log to file, but rotate daily
    new DailyRotateFile({
      // each file name includes current date
      filename: "logs/rotating-logs-%DATE%.log",
      datePattern: "MMMM-DD-YYYY",
      zippedArchive: false, // zip logs true/false
      maxSize: "20m", // rotate if file size exceeds 20 MB
      maxFiles: "14d", // max files
    }),
  ],
});

The function above shows how to create a logger instance, the format configuration of what you want to log and how it should look like, and the transport which either prints to the console when on development or test environment.It also saves to file using the DailyRotateFile method from the winston-daily-rotate-file package. To learn more about how to set up Winston for logging check out this documentation.

Next, you will write the function that formats your logs to readable JSON. This is where you will call the function that sends the alert to WhatsApp when certain conditions are met. You will also use the function that redacts sensitive information here to remove them from the formatted logs. Create a new file in the src/utils/log directory named formatLog.js and add the following code to it:

import { sendWhatsAppAlert } from "../alert/alertFunctions.js";
import { HTTPHeaders } from "../constants/constants.js";
import { redactLogData } from "./redactedData.js";

const formatHTTPLoggerResponse = (req, res, responseBody, requestStartTime) => {
  let requestDuration = "";
  let startTime = "";
  const formattedBody = JSON.parse(responseBody);
  const textBody = {
    request: {
      host: req.headers.host,
      url: req.url,
      body: (req.body && redactLogData(req.body)) || {},
      params: req?.params,
      query: req?.query,
      clientIp:
        req?.headers[HTTPHeaders.ForwardedFor] ?? req?.socket.remoteAddress,
    },
    response: {
      statusCode: res.statusCode,
      requestDuration,
      body: redactLogData(formattedBody),
    },
  }

  if (requestStartTime) {
    const endTime = Date.now() - requestStartTime;
    requestDuration = `${endTime / 1000}s`; // ms to seconds
    // Create a Date object from the timestamp
    const date = new Date(requestStartTime);
    // Format the date into a human-readable string
    startTime = date.toLocaleString();
  }

  // message param for twilio alert
  const messageParams = {
    errorDescription: formattedBody?.message,
    affectedEndpoint: req.baseUrl,
    startTime: startTime,
    duration: requestDuration,
    details: redactLogData(textBody),
    alertType: res.statusCode >= 500 ? "Critical" : "Error",
    method: req.method,
  };

  if (res.statusCode >= 400) {
    sendWhatsAppAlert(messageParams);
  }
  return {
    request: {
      headers: (req.headers && redactLogData(req.headers)) || {},
      host: req.headers.host,
      baseUrl: req.baseUrl,
      url: req.url,
      method: req.method,
      body: (req.body && redactLogData(req.body)) || {},
      params: req?.params,
      query: req?.query,
      clientIp:
        req?.headers[HTTPHeaders.ForwardedFor] ?? req?.socket.remoteAddress,
    },
    response: {
      headers: res.getHeaders(),
      statusCode: res.statusCode,
      requestDuration,
      body: redactLogData(formattedBody),
    },
  };
};

export default formatHTTPLoggerResponse;

A few things to note about the code above:

  • The responseBody is being parsed as a normal object to a JSON object.
  • The textBody variable stores the details of the log we will share through WhatsApp.
  • The messageParams are the parameters used in calling the sendWhatsAppAlert function to work
  • All the message body parameters are parsed through the redactLogData function to remove sensitive data.
  • If the status code is greater or equal to 400, it triggers the WhatsApp alert.
  • The alert is tagged Critical for status code 500 and above and Error for status code 400 and below 500.

Lastly, for the logging functionalities, you will create a file that intercepts all requests that happen in the application to pick up the logs. It will be used in the entry server file as a middleware above where the routes that it should intercept are defined or instantiated. This file will be created in the src/utils/log directory and named interceptor.js. Add the following code to the file:

import formatHTTPLoggerResponse from "./formatLog.js";
import { HTTPMethods, SuccessMessages } from "../constants/constants.js";
import { httpLogger } from "../../middlewares/logger.js";

export const responseInterceptor = (req, res, next) => {
  // used to calculate time between request and the response
  const requestStartTime = Date.now();
  // Save the original response method
  const originalSend = res.send;

  let responseSent = false;

  // Override the response method
  res.send = function (body) {
    if (!responseSent) {
      if (res.statusCode < 400) {
        httpLogger.info(
          getResponseMessage(req.method),
          formatHTTPLoggerResponse(req, res, body, requestStartTime)
        );
      } else {
        httpLogger.error(
          body.message,
          formatHTTPLoggerResponse(req, res, body, requestStartTime)
        );
      }

      responseSent = true;
    }

    // Call the original response method
    return originalSend.call(this, body);
  };

  // Continue processing the request
  next();
};

function getResponseMessage(responseMethod) {
  switch (responseMethod) {
    case HTTPMethods.POST:
      return SuccessMessages.CreateSuccess;
    case HTTPMethods.GET:
      return SuccessMessages.GetSuccess;
    case HTTPMethods.PUT || HTTPMethods.PATCH:
      return SuccessMessages.UpdateSuccess;
    case HTTPMethods.DELETE:
      return SuccessMessages.DeleteSuccess;
    default:
      return SuccessMessages.GenericSuccess;
  }
}

In this code, the incoming request and response body are picked by the middleware, the original res.send method is stored and the body is overridden to be used as the log data first then reset back to the original before it continues to the next thing which is sending the original res.send body to the client.

The getResponseMessage helper function does match the HTTP method in the response with the right message saved in the src/utils/constant/constant.js file. The logs are captured but first parsed through the formatHTTPLoggerResponse function we created earlier to get a formatted JSON object.

Updating the Server File

You will proceed to add the middleware to the server file which is our entry to the app. The file is located in the root directory as index.js. You will add the middleware before the route definitions so you can intercept the requests that will go through them. Update the code with the one below:


import express from "express";
import cors from "cors";
import redis from "redis";
import bodyParser from "body-parser";
import { port, redisURL } from "./src/config/index.js";
import { AppError } from "./src/utils/responseHandler.js";
import { responseInterceptor } from "./src/utils/log/interceptor.js";
import swaggerDocs from "./swagger.js";
import "./src/models/index.js";
import userRouter from "./src/routes/users.js";
import authRouter from "./src/routes/auth.js";

const app = express();

app.use(cors());
app.use(express.json());
app.disable("x-powered-by"); // less hackers know about our stack
app.use(bodyParser.urlencoded({ extended: false }));

// Your middleware function to handle errors
const errorHandler = (err, req, res, next) => {
  if (res.headersSent) {
    return next(err);
  }
  if (err instanceof AppError) {
    // If it's a CustomError, respond with the custom status code and message
    return res
      .status(err.statusCode)
      .json({ status: err.status, error: err.message, code: err.statusCode });
  } else {
    // If it's an unknown error, respond with a 500 status code and a generic error message
    return res
      .status(500)
      .json({ status: "critical", error: "Internal Server Error.", code: 500 });
  }
};

// Applying the error handling middleware
app.use(errorHandler);

// create a client connection
export const client = redis.createClient({
  url: redisURL,
});
// on the connection
client.on("connect", () => console.log("Connected to Redis"));
client.connect();

// Run the swagger docs before log interception
swaggerDocs(app, port);

// Place an interceptor above all routes that you want to `intercept`
app.use(responseInterceptor);

/** HTTP GET Request */
app.get("/", (req, res) => {
  res.status(201).json("Home GET Request");
});
app.use("/api/v1/user", userRouter);
app.use("/api/v1/auth", authRouter);

app.listen(port, () => {
  console.log(`
      ###########################################
      Server is currently running at port ${port}
      ###########################################`);
});

Testing and Product Demonstration

Your app is now ready for testing. Before you start testing on WhatsApp, you can proxy the localhost server with ngrok to the internet by running the below command on another tab on your terminal.

ngrok http 5000

Remember having ngrok installed and authenticated is one of the prerequisites to starting this tutorial.

You will get a response like this with your public app address hosted on ngrok. Your server is now up and running.

Terminal output after running ngrok tunnel

Once the app is running, open your WhatsApp and send join <sandbox code>  first in order to establish a connection to the sandbox. When you have established a connection, you can go ahead to make a sample request to your API. In this case, I created a user and tried to log in with the wrong credentials to get the alert. Below is a demonstration of how it works:

  • Navigate to ngrok Forwarding URL and append /docs to open up the Swagger UI. Register a user using the /signup endpoint on the /api/v1/auth route

Swagger UI showing user registration via the register endpoint
  • Sign in or Log in using the wrong credentials to get a 400 status code. If it fails, try to copy the shown curl request on another tab on your terminal.

Swagger UI showing user login with wrong credential via the login endpoint
  • Get the error alert on the WhatsApp number set as to in the Twilio function.

Incident alert notification using Twilio WhatsApp API

Conclusion

If you followed till this point, congratulations! You have been able to use Twilio's powerful communication suites to build a handy incidence alert service. This service can be extended to many other possibilities like getting daily summaries on each endpoint with regard to your API health etc. You can also automate /slash commands that create a GitHub issue for any incident critical enough. You can learn more about using Twilio WhatsApp API in a production environment by referring to Twilio’s documentation.

Desmond Obisi is a software engineer and a technical writer who loves developer experience engineering. He’s very invested in building products and providing the best experience to users through documentation, guides, building relations and strategies around products. He can be reached on Twitter, LinkedIn or through my mail desmond.obisi.g20@gmail.com.

 

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.