Build Generative AI Email Experiences using SendGrid Inbound Parse

Build Generative AI Email Experiences using SendGrid Inbound Parse

Agentic Email with Twilio SendGrid? Yes, Please! Part 2: Process Inbound Emails with AI

In this post, I’ll show you how to process inbound emails you gather with Twilio SendGrid’s Inbound Parse and respond with Generative AI, letting you build an agentic experience all over email.

In Part 1 of this series, we covered how to receive and process inbound emails (and attachments) with SendGrid’s Inbound Parse. At the end of that tutorial, you had the components of your email in an S3 bucket, and had triggered an SNS message with the details needed to find that bucket. In this post, I’ll show you how to take those components and formulate and send a message with Generative AI.

Let’s get started.

GenAI email agent experience architecture

If you didn’t follow along with part 1 of the series, this solution is broken into two parts:

1. Inbound Parse with Signature Verification
2. Process Inbound Emails with Generative AI

In Part 2, here is what you’ll be building:

Let’s start from the top…

1. Input: The down arrow is an SNS message that gets published once an inbound email has been completely checked and processed by the systems deployed in Part 1. This message lets downstream processors know that there is more work to do.
2. First Pass: This lambda receives the SNS message and retrieves the content of the email from the S3 bucket using the messageId as the key. The content is then sent to an LLM model via AWS Bedrock. The prompt asks the model to summarize the email and categorize it as Sales, Support, Account, Inquiry, or Conversation. The results are saved in the JSON file, and the lambda publishes an event to an EventBus to route to the next step.
3. If there are Attachments: If there are attachments, the event is delivered to a lambda that can analyze different types of files. The lambda loads the attachments from the S3 bucket, then submits them to the LLM to generate a summary. The summary is added to the JSON file, and another event is published to the EventBus
4. Category Handlers: The event is delivered to one of 5 category handlers depending on the category set in the First Pass lambda. Each of these lambdas have their own prompts and business logic. They are all basic, but you should be able to see how you could build specific handlers based on the available context.
5. Trigger additional Events: The category handlers can trigger additional events (send a reply email, connect to a human agent, update a CRM or support ticket, …). For example, one trigger you could add is to send additional outbound emails in response to the inbound email. This repo attached to this blog post also contains a workflow that sends outbound emails using Twilio SendGrid.

Prerequisites

As said in part 1, this is not a beginner level build. Since you must complete Part 1 as a prerequisite to Part 2, you should have already completed all of the prerequisites and are ready to go!

Return to Part 1 if you have not deployed that yet… here are the services you’ll need to set up before you continue:

• Twilio SendGrid Account. If you don’t yet have one, you can sign up for a free account here.
• A domain that you can verify and access DNS records.
• You can learn how to verify a domain with this tutorial.
• AWS Account with permissions to provision API Gateways, Lambdas, S3 buckets, IAM Roles & Policies, Bedrock, SQS, EventBridge, and SNS topics. You can sign up for an account here.
• AWS CLI installed with AWS credentials configured.
• AWS SAM CLI installed.
• Node.js installed on your computer.

Clone the repo and build the app

The instructions to spin up Part 2 are in the repo. Please proceed to the:

🚀 README for Part 2

Run and test the Generative AI email experience

Do you remember we tested our setup by sending an email in Part 1? Let’s send some test emails – or continue from the example in Part 1 – to explore how we can use Generative AI to craft an email experience, now that we know we can receive inbound email and attachments.

Example 1: Process an email inquiry (from Part 1)

Send an email

Remember the email that we sent in Part 1? If you copied my example, I sent the following message:

I would like to purchase one of your products for my dog. Can you make some suggestions? I attached a photo of my dog.

I then attached an (awesome) picture of my dog, Mabel.

Let’s see what part 2 did with it…

LLM response

This email was categorized as an inquiry. The inquiry handler took it and crafted a response that included the image attachment. The prompt used to summarize and categorize inbound emails can be viewed in the repo. The prompt is general but will give you a place to start. In production, you will of course enter your own rules to match your business requirements and to anticipate the type of emails you expect to receive and how to handle and identify edge cases.

Here is the JSON:

Triggered event

The inquiry handler triggers an event to send an email reply via SendGrid.

Based on the JSON in the above step, here is what was sent:

Pretty awesome! The AI response seemed to do a great job recommending products to a user. Now, let’s try handling something a little more difficult, and reach out with a problem.

Example 2: Open a support ticket

Let’s say that you finally saved up to purchase an action figure that you’ve always wanted. But when it arrived it was broken!

Let’s test the setup by emailing and asking for help with a broken action figure.

Initial email

Craft a message to support stating the issue you experienced. Like with our first example, let’s test the system by again sending an attachment to the AI.

Here is my email to support you can copy:

I need help! I purchased this from you guys and when it arrived it was broken! What should I do? Photo below…

Again, send this email to the domain you configured with Inbound Parse. This email will go to SendGrid, and be routed to your Inbound Parse webhook. The security signature will then be checked, and the email contents stored (like with our product inquiry email), then summarized and routed. It should ultimately end up at the specialized Support Handler where next steps are determined and executed.

Reply back from Support

The steps here are similar to the LLM response and Triggered event step above, except you can see they were handled with a different route.

Notice that the photo attachment was analyzed so the support handler was able to pass context about the contents of the photo – it knew that it contained a broken action figure.

In addition, the other red arrow shows how the support handler injected a ticket number in the reply. Let’s exercise the workflow and reply to this message and build the support case.

Reply back to the GenAI support response

You want to be sure that you do not have to pay for any of this… right? Send an email back clarifying how the replacement shipment will be handled.

To copy my message, here is what I wrote:

Ok – will you pay for everything? I want to be sure I don’t have to pay any more for this item. I was really looking forward to having this today.

Reply, and let’s trace what happens next.

Receive a threaded response back from support

If everything went right, the support handler will get this reply because it noticed the ticket number in the subject line. The full context is used for the reply back.

This example introduced a method for tracking email threads by using a ticket number in the subject line. Here are a few ways you can ensure continuity of messages:

Use a Reference Number in the Subject Line: This method is relatively simple (and this example shows how well it works!). It supports additional people to be added via cc. However, the ticket number in the subject line is visible to the customer and it could be overwritten, which means a production solution will need to be coded defensively.

Using a Dynamic Reply-To: Instead of injecting an ID into the subject line, some of our customers choose to set the REPLY-TO address in their email to something that includes a unique ID. For example, support_12345ABCD@your-domain.com. This is easy to implement and scalable, and more robust to being overwritten – if a user changes the address, the email won’t be delivered, and they should double check the address

Using References in the HTML Content: IDs can be “hidden” in the body of the message and retrieved via text search. This is likely less reliable (consider that a user might “Select all” and delete) but could be used as an additional method of tagging an email to a specific ID.

Internally Tracking: Some of our customers build their own systems to track email threads by using TO, FROM, SUBJECT, and TIMESTAMP to generate a unique key. How you implement this will depend on your business requirements and how you and your customers interact over email.

Use multiple methods: Many of our customers will use more than one of the methods listed above to ensure.

Other examples

We have covered a general inquiry and a support use case. This repository also has 3 other handler types.

• Sales → Will be triggered if the inbound message is related to purchasing.
• Account → Will be triggered if the inbound message wants to make changes to their account.
• Conversation → This use case just shows how it is possible to have a conversation over email. The conversation messages are stored in S3 to persist the chat history.

These use cases are basic and intended to give you any idea of what you could build! You should be able to see how you could customize all parts of the process to meet your specific business needs.

Deploy to production

While you can get this system working pretty quickly, it is not ready for your production environment. This blog post and repo is intended to inspire and help you start building awesome applications.

Conclusion

In the two steps of this tutorial, you’ve seen how to receive emails (and attachments) with SendGrid’s Inbound Parse, then send Generative AI-driven responses using AWS Bedrock. You can now build the Agentic Email experience of your – and your customers’ – dreams.

You now have a plan to help you start building! Let us know what you build so we can try it out.

Dan Bartlett has been building web applications since the first dotcom wave. The core principles from those days remain the same but these days you can build cooler things faster. He can be reached at dbartlett [at] twilio.com.