Open and Click Tracking for Email using Java and Twilio SendGrid

Time to read: 5 minutes

April 23, 2020
Written by
Matthew Gilliard
Twilion

Open and Click Tracking for Email using Java and Twilio SendGrid

I covered sending emails from Java using the Twilio SendGrid API before, but for important emails it can be vital to know whether recipients have opened your mail, and whether they have clicked on the links in it. You can do this using the SendGrid Event Webhook, which works by sending HTTP requests to a URL you provide. In this post I’ll show how to configure this with Twilio SendGrid and build a Java web application to handle the webhooks and record your recipients’ activity.

Before you start

You will need

  • Java - version 8 or newer. I like to use SDKMAN! to manage Java installations
  • Ngrok
  • A Twilio SendGrid account - sign up here if you don’t have one already.

Creating the application

If you prefer to skip the coding and instead try things out you can find the code for this application on GitHub, and should skip ahead to the section below called Creating a public URL for your application with ngrok.

The application will be built with Spring Boot and Spring MVC annotated controllers. When Twilio SendGrid events are received by the application, it will store open and click events in a couple of Map objects to record which addresses have opened a particular email or clicked on a particular link. There are also a couple of endpoints so that you can retrieve that data, for example to use in a dashboard.

Creating the application template

You can use SDKMAN! to install the Spring Boot cli tool with sdk install springboot. Use this to create a new project in an empty directory with:

spring init \
  --dependencies web \
  --build maven \
  --groupId lol.gilliard \
  --artifactId sendgrid-event-hooks \
  --extract

This command sets up a new Spring Boot project in your current directory. Feel free to choose your own groupId and artifactId using the Maven naming conventions. If you don’t want to use the CLI tool, you can enter the same details at start.spring.io then download and unzip the project which is generated.

You can now import the project into your IDE and start coding. My favourite is IntelliJ IDEA but Eclipse and NetBeans are also popular.

Adding the events controller

In the same package as the Application class created by the Spring Initializr, create a class called SendGridEvent which will hold a subset of the JSON data posted by SendGrid. Spring will take care of turning the JSON into instances of this class.

That class will have a few final fields, so it will need a constructor but we can do without getters and setters:

import com.fasterxml.jackson.annotation.JsonProperty;

public class SendGridEvent {

   public final String email;
   public final String eventType;
   public final String url;
   public final String sgMessageId;

   public SendGridEvent(@JsonProperty("email") String email,
                        @JsonProperty("event") String eventType,
                        @JsonProperty("url") String url,
                        @JsonProperty("sg_message_id") String sgMessageId) {
       this.email = email;
       this.eventType = eventType;
       this.url = url;
       this.sgMessageId = sgMessageId;
   }
}

The @JsonProperty annotations allow the fields in this class to be named differently from the fields in the JSON.

In the same package, create a class called SendGridEventWebhookHandler annotated with @Controller which has methods for receiving events by POST requests and serving up the data by GET requests:


import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.*;

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.logging.Logger;

@Controller
public class SendGridEventWebhookHandler {

   private static final Logger logger = Logger.getLogger(SendGridEventWebhookHandler.class.getName());

   private final Map<String, Set<String>> openedEmails = new ConcurrentHashMap<>();
   private final Map<String, Set<String>> clickedLinks = new ConcurrentHashMap<>();

   @PostMapping("/events")
   @ResponseBody
   public String receiveSGEventHook(@RequestBody List<SendGridEvent> events) {

       logger.info(String.format("Received %d events", events.size()));

       events.forEach(event -> {

           switch (event.eventType) {
               case "open":
                   openedEmails.computeIfAbsent(event.sgMessageId, k -> new HashSet<>()).add(event.email);
                   break;

               case "click":
                   clickedLinks.computeIfAbsent(event.url, k -> new HashSet<>()).add(event.email);
                   break;
           }
       });

       return "ok";
   }


   @GetMapping("/opened")
   @ResponseBody
   public Map<String, Set<String>> getOpenedEmailData(){
       return openedEmails;
   }

   @GetMapping("/clicked")
   @ResponseBody
   public Map<String, Set<String>> getClickedLinksData(){
       return clickedLinks;
   }

}

The receiveSGEventHook (highlighted) method takes a List<SendGridEvent> parameter which will be created from the JSON sent by SendGrid. It is a List because SendGrid can batch up events if many arrive in a short space of time.

openedEmail is a Map which maps sg_message_id to a Set of email addresses which have opened the email with that ID. This id can be looked up by performing an “advanced search” in your Activity Feed and filtering by Message ID. The Message ID may be different for each recipient. If you prefer to use your own categorization system for emails, then the Unique Arguments feature is designed for that purpose.

clickedLinks is a similar structure which maps URLs to a Set of email addresses whose owner has clicked on the link. In both cases I have used ConcurrentHashMap and computeIfAbsent to avoid thread-safety problems if multiple sets of events are received simultaneously.

When events are received the code loops through them using forEach and adds any open or click events to the appropriate Map. There are several other types of events such as bounce and unsubscribe, which this application ignores. Later you will see how to configure which types of events SendGrid will send.

The final thing needed is a couple of methods to expose the data from the openedEmails and clickedLinks maps. I added an endpoint for each:

@GetMapping("/opened")
@ResponseBody
public Map<String, Set<String>> getOpenedEmailData(){
   return openedEmails;
}

@GetMapping("/clicked")
@ResponseBody
public Map<String, Set<String>> getClickedLinksData(){
   return clickedLinks;
}

This is all the code needed for the application. Run it from the terminal with ./mvnw spring-boot:run in the root directory of the project.

Creating a public URL for your application with ngrok

So far, the application is running and listening for HTTP requests on your localhost, but SendGrid needs a public URL for the webhook. You could deploy this application on a cloud server, but to keep it simple here I recommend using ngrok, which can create public URLs that are tunneled to your localhost server.

After installing ngrok, create a tunnel with ngrok http 8080:

Screenshot of ngrok output highlighting the https forwarding url

Take note of the https forwarding URL as that will be needed when configuring SendGrid. From the screenshot above, my webhook URL will be https://3b7e5043.eu.ngrok.io/events - your ngrok domain will be different, but don’t forget to add /events at the end.

Enable Event Webhooks & set the webhook URL

Once you have created your SendGrid account, there are three things to do:

  • Add open-tracking and click-tracking to your outbound emails
  • Configure SendGrid with your ngrok URL for webhooks
  • Configure SendGrid to make webhook requests for open and click events.

Adding open-tracking and click-tracking to outbound emails

On the Tracking Settings section of your SendGrid dashboard, enable “Open Tracking” and “Click Tracking”

Screenshot of "Tracking Settings". Open Tracking and Click Tracking are both enabled.

Open Tracking configures SendGrid to add an invisible image to each outbound email which is used to tell whether the recipient has opened the email. Click Tracking configures SendGrid to rewrite URLs in your messages to track which recipients have clicked on which URLs. Make sure to read these best practices for how to construct links to make sure they will be tracked.

Configure SendGrid to use your application for webhooks

From the Mail Settings page on your dashboard, enable Event Webhooks, and configure the HTTP Post URL to be your ngrok URL followed by /events:

Event Webhook configuration. Setting the HTTP Post URL

Configure SendGrid to send Open and Click events

While you are on the Event Webhook configuration page, check the boxes for Opened and Clicked under Engagement Data:

Event Webhook configuration. Selecting "Opened" and "Clicked" events to be posted.

Finally, enable the Event Webhook and save the configuration

Event Webhook configuration. Enabling and Saving the webhook config

Testing your application from the SendGrid dashboard

On the Event Webhook configuration page there is a button to test your integration, which you can use now. You will see the following in your Spring Boot application logs when you click it:

l.g.s.SendGridEventWebhookHandler            : Received 11 events

You can check that the data from these test events has been stored in your application by making an HTTP request to localhost:8080/opened and localhost:8080/clicked either in your browser or using a tool like curl or (my favourite) HTTPie.

Example JSON in the browser for `/opened` endpoint

Example JSON in the browser for `/clicked` endpoint

Using your application for real

The instructions from How to Send Email in Java using Twilio SendGrid can be used to test this with a single change - you need to add a link to the email to check click-tracking. Remember to follow the Click Tracking Best Practises to make sure that your URLs will be tracked correctly. The code to send email will look like this:

import com.sendgrid.*;
import com.sendgrid.helpers.mail.Mail;
import com.sendgrid.helpers.mail.objects.*;

import java.io.IOException;

public class SendGridEmailer {

   public static void main(String[] args) throws IOException {

       Email from = new Email("test@example.com");
       Email to = new Email("matthew@<REDACTED>"); // use your own email address here

       String subject = "Sending with Twilio SendGrid is Fun";
       Content content = new Content("text/html", "and <em>easy</em> to do anywhere with <strong>Java</strong>. Here is <a href=\"https://www.sendgrid.com\">a link</a>");

       Mail mail = new Mail(from, subject, to, content);

       SendGrid sg = new SendGrid(System.getenv("SENDGRID_API_KEY"));
       Request request = new Request();

       request.setMethod(Method.POST);
       request.setEndpoint("mail/send");
       request.setBody(mail.build());

       Response response = sg.api(request);

       System.out.println(response.getStatusCode());
       System.out.println(response.getHeaders());
       System.out.println(response.getBody());
   }
}

You can add that class to your project and run it from the IDE while the Spring Boot app is still running in the terminal. Just make sure that you add the sendgrid-java dependency to your build as described in that post.

When you run the main method in that class, you will send yourself an email. After you open it in your mail client, SendGrid will report an open event, and when you click the link there will be a click event. These might take a minute or so to arrive, and might be batched together but your code will handle that just fine.

And, voila! You can now track which of your recipients has opened your emails and clicked the lovely links they contain.

Wrapping up

If you are building with Twilio or SendGrid, I’d love to hear about it - let me know.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.