Apache Kafka destination

(new)

Beta

Data Destinations is currently available as a Private Beta product and the information contained in this document is subject to change. Data Destinations is not a HIPAA Eligible Service or PCI compliant and should not be enabled in workflows that are subject to HIPAA or PCI. To learn more, see Private beta legal information.

Apache Kafka is a distributed event-streaming platform used for high-throughput, fault-tolerant message delivery. This destination publishes your events directly to a Kafka topic you manage on a publicly reachable provider, such as Confluent Cloud or Amazon MSK. Self-hosted clusters on a private network aren't supported.

Destination overview

Category: Raw data
Release status: Beta
Authentication: SASL
Regional availability: US1

Join the Data Destinations private beta

Get early access to native destinations for data warehouses, observability tools, and event pipelines.

Eligible events

You can send all event types supported by Event Streams to Apache Kafka. This destination has no user-identifier restriction.

Get started

Prerequisites

Before you set up a Kafka destination, make sure you have:

A publicly reachable Kafka cluster (self-hosted clusters on a private network aren't supported) with a topic already created. Twilio doesn't create topics automatically, so delivery to a missing topic fails.
The broker addresses for your cluster (for example, broker1:9092,broker2:9092).
A SASL username and password for a SASL mechanism your cluster supports (PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512). This destination requires SASL authentication, and you can't currently configure it without credentials.
Describe and Write permissions on the target topic for that SASL principal, if your cluster enforces ACLs. Twilio uses Describe for the connectivity check it runs when you save and Write to publish events.
TLS turned on for the broker listeners you're connecting to, if your cluster requires TLS.

Create the destination

(information)

Info

See Set up a destination for the full UI and Destinations API setup flow common to every destination type.

Twilio Console

To create an Apache Kafka destination in the Twilio Console:

From your Kafka cluster, gather the settings listed in Destination settings.
In the Twilio Console, go to Data > Data Destinations > Destinations > Add destination subscription.
Complete the Add destination subscription wizard.

Destinations API

To create an Apache Kafka destination through the Destinations API, send a POST request to the Destinations endpoint and set destinationType to kafka. For authentication and the other request body fields, see Create a destination.

Create a destination

POST "https://destinations.twilio.com/v1/ControlPlane/Destinations"

Set the config object in the request body as follows:

Name	Type	Description
`brokers` (Required)	string	A comma-separated list of broker addresses, for example `broker1:9092,broker2:9092`.
`credentials.credentialType` (Required)	string	Set to `SASL`.
`credentials.password` (Required)	string	SASL password.
`credentials.saslMechanism` (Required)	enum`<string>`	The SASL authentication mechanism. Valid values: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`.
`credentials.username` (Required)	string	SASL username.
`tlsEnabled`	boolean	Whether to use TLS for broker connections.
`tlsSkipVerify`	boolean	Whether to skip TLS certificate verification. Not recommended for production.
`topic` (Required)	string	Target Kafka topic name.

The following example shows a request body. Replace the uppercase placeholder values with your own.

Example request body

1{
2  "destinationType": "kafka",
3  "name": "My Apache Kafka destination",
4  "config": {
5    "brokers": "BROKERS",
6    "credentials": {
7      "credentialType": "SASL",
8      "password": "PASSWORD",
9      "saslMechanism": "PLAIN",
10      "username": "USERNAME"
11    },
12    "topic": "TOPIC"
13  }
14}

Destination settings

Use kafka as the destinationType.

Field	Description	Required	Default or choices
Brokers	Comma-separated broker addresses (for example, `broker1:9092,broker2:9092`)	Yes	None
Topic	Target Kafka topic name. Must already exist, since Twilio doesn't create it for you.	Yes	None
SASL Mechanism	SASL authentication mechanism	Yes	`PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`
Username	SASL username	Yes	None
Password	SASL password (sensitive, stored encrypted)	Yes	None
TLS Enabled	Encrypt the connection to your brokers	No	`false`
TLS Skip Verify	Skip TLS certificate verification (not recommended for production)	No	`false`

Twilio authenticates to your Kafka cluster by using SASL (PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512), optionally combined with TLS encryption. SASL Mechanism, Username, and Password are all required, since you can't currently configure this destination for a cluster that doesn't require credentials.

Default mappings

Field	Kafka field
Event source	Partition key
Event payload	Message value (JSON)

Verification and troubleshooting

When you save a Kafka destination, Twilio validates connectivity by issuing a non-destructive metadata request to your brokers. This request proves TCP reachability and SASL authentication without publishing any data. If validation fails, check that:

Your Brokers address is correct and reachable from the public internet (or your configured network path).
Your SASL Mechanism, Username, and Password are correct.
TLS Enabled matches what your cluster listener actually requires.

Twilio doesn't currently offer a way to manually trigger a test event for this destination, from either the Console or the CLI. To confirm delivery works, wait for a real event of a subscribed type to occur.

Common errors

Cause	What you'll see	Fix
Broker unreachable	Validation fails with a connection error	Confirm the broker address and network path (firewall, security group, VPC peering)
Invalid SASL credentials	Validation fails with an authentication error	Re-check your username, password, and SASL Mechanism
SASL principal lacks `Describe` ACL on the topic	Validation fails with an authorization error, even with correct credentials	Grant the SASL principal `Describe` on the topic
SASL principal lacks `Write` ACL on the topic	Passes validation, then delivery errors reported in the Debugger	Grant the SASL principal `Write` on the topic
Topic does not exist	Delivery errors reported in the Debugger	Create the topic on your cluster. Twilio doesn't create topics automatically
TLS mismatch	Connection or handshake errors	Confirm TLS Enabled matches your broker listener's configuration
Event payload exceeds the broker's `message.max.bytes`	Delivery errors reported in the Debugger; not retried	Reduce the event payload size, or raise `message.max.bytes` on your broker/topic

If Twilio can't deliver events to your destination, Twilio sends an error notification to the Debugger (available in US1 region only). If Twilio continues to encounter delivery errors, Twilio notifies you every 20 minutes after the initial error.