Databricks destination

(new)

Beta

Data Destinations is currently available as a Private Beta product and the information contained in this document is subject to change. Data Destinations is not a HIPAA Eligible Service or PCI compliant and should not be enabled in workflows that are subject to HIPAA or PCI. To learn more, see Private beta legal information.

Databricks is a data lakehouse platform for analytics, ML, and data warehousing. Sending events to Databricks lets you stream communications data directly into a Delta table for low-latency SQL analytics and downstream ML.

Twilio delivers events using Zerobus Ingest, Databricks' push-based streaming service that writes records directly into a Unity Catalog Delta table with no intermediate broker. Twilio streams each event straight into your table over a gRPC connection, and the event is typically queryable within seconds. Check Databricks' ingestion availability documentation to confirm Zerobus Ingest is available in your workspace's region before setting up this destination.

Destination overview

Category: Warehouse
Release status: Beta
Authentication: OAuth (client ID and secret)
Regional availability: US1

Join the Data Destinations private beta

Get early access to native destinations for data warehouses, observability tools, and event pipelines.

Eligible events

You can send all event types supported by Event Streams to Databricks. This destination has no user-identifier restriction.

Get started

Prerequisites

Before you set up a Databricks destination, make sure you have:

A Databricks workspace on Amazon Web Services, Azure, or Google Cloud.
A Databricks service principal with an OAuth Client ID and Client Secret (found in your Databricks account under Service Principals).
An existing Unity Catalog catalog and schema that the service principal has USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY, and SELECT privileges on. CREATE TABLE is needed for initial provisioning of the destination table (twilio_events_<subscription ID>), and MODIFY/SELECT are needed for ongoing Zerobus Ingest writes and reads. The catalog and schema themselves must already exist. Twilio creates only the table.
A SQL warehouse for the service principal to use when Twilio creates the destination table.

Create the destination

(information)

Info

See Set up a destination for the full UI and Destinations API setup flow common to every destination type.

Twilio Console

To create a Databricks destination in the Twilio Console:

From your Databricks account, gather the settings listed in Destination settings.
In the Twilio Console, go to Data > Data Destinations > Destinations > Add destination subscription.
Complete the Add destination subscription wizard.

Destinations API

To create a Databricks destination through the Destinations API, send a POST request to the Destinations endpoint and set destinationType to databricks. For authentication and the other request body fields, see Create a destination.

Create a destination

POST "https://destinations.twilio.com/v1/ControlPlane/Destinations"

Set the config object in the request body as follows:

Name	Type	Description
`catalog` (Required)	string	Unity Catalog catalog that holds the target tables.
`credentials.credentialType` (Required)	string	Set to `CUSTOM`.
`credentials.clientId` (Required)	string	The application ID (UUID) of your OAuth machine-to-machine service principal.
`credentials.clientSecret` (Required)	string	The OAuth secret generated for the service principal.
`schema` (Required)	string	Unity Catalog schema (inside catalog) that holds the target tables.
`warehouseId` (Required)	string	The SQL warehouse that creates the per-subscription Delta table when you save a subscription.
`workspaceUrl` (Required)	string	Your Databricks workspace URL, for example `https://dbc-abc123.cloud.databricks.com`.
`zerobusEndpoint` (Required)	string	The Zerobus gRPC ingest endpoint host, in the format `WORKSPACE_ID.zerobus.REGION.cloud.databricks.com`.

The following example shows a request body. Replace the uppercase placeholder values with your own.

Example request body

1{
2  "destinationType": "databricks",
3  "name": "My Databricks destination",
4  "config": {
5    "catalog": "CATALOG",
6    "credentials": {
7      "credentialType": "CUSTOM",
8      "clientId": "CLIENT_ID",
9      "clientSecret": "CLIENT_SECRET"
10    },
11    "schema": "SCHEMA",
12    "warehouseId": "WAREHOUSE_ID",
13    "workspaceUrl": "WORKSPACE_URL",
14    "zerobusEndpoint": "ZEROBUS_ENDPOINT"
15  }
16}

Destination settings

Use databricks as the destinationType.

Field	Description	Required	Default or choices
Client ID	OAuth service-principal Application ID (UUID). Sensitive, stored encrypted.	Yes	None
Client Secret	OAuth secret generated for the service principal. Sensitive, stored encrypted.	Yes	None
Workspace URL	Your Databricks workspace URL. For example, `https://dbc-abc123.cloud.databricks.com`.	Yes	None
Zerobus Endpoint	The Zerobus gRPC ingest endpoint host for your workspace's region, in the form `<workspace-id>.zerobus.<region>.cloud.databricks.com`.	Yes	None
Catalog	The Unity Catalog catalog that holds the target table. You can't change this after creating the destination.	Yes	None
Schema	The Unity Catalog schema (inside Catalog) that holds the target table. You can't change this after creating the destination.	Yes	None
SQL Warehouse ID	The SQL warehouse Twilio uses to create the destination table for you.	Yes	None

Twilio authenticates to Databricks by using OAuth 2.0 machine-to-machine (client credentials) with your service principal's Client ID and Client Secret. Twilio exchanges these for a short-lived access token and sends it as a bearer token on every request. Twilio never sends your Client Secret itself to Zerobus.

The SQL Warehouse ID and Zerobus Endpoint serve two different purposes. Twilio uses the SQL warehouse once when accessing the SQL Statement Execution API, to create the destination table the first time you save a subscription. Twilio streams every event after that directly into the table over Zerobus Ingest. The SQL warehouse isn't involved in ongoing delivery.

The first time you save a subscription, Twilio creates the destination table twilio_events_<subscription ID> in your configured catalog and schema.

Default mappings

Field	Databricks column
Event ID	`event_id`
Event type	`event_type`
Event time	`event_timestamp`
Event payload	`payload`

Verification and troubleshooting

When you save a Databricks destination, Twilio validates your Client ID and Client Secret by requesting an OAuth token from your workspace. If validation fails, check that:

Your Client ID and Client Secret are correct and haven't been revoked in Databricks.
Your Workspace URL is correct and reachable.

This validation confirms your credentials only. It doesn't confirm that the service principal has the required Unity Catalog grants or that your catalog and schema exist. If your principal is missing the required Unity Catalog grants or your catalog and schema don't exist, table creation or event delivery might fail after you save the destination. If events aren't arriving as expected, check your service principal's grants on the target catalog and schema.

Twilio doesn't currently offer a way to trigger a test event for this destination from either the Console or the Twilio command line interface (CLI). To confirm delivery is working, wait for a real event of a subscribed type to fire.

Common errors

Cause	What you'll see	Fix
Invalid or revoked Client ID / Client Secret	Validation fails with an auth error	Re-check the credentials in Databricks and update the destination
Service principal missing Unity Catalog grants	Table creation or delivery fails after the destination validates	Grant `USE CATALOG`, `USE SCHEMA`, and `CREATE TABLE`/`MODIFY`/`SELECT` to the service principal on your catalog and schema
Wrong region in Zerobus Endpoint	Delivery errors reported in the Debugger	Confirm the endpoint's region matches your workspace's region
Zerobus Ingest isn't available in your workspace's region	Delivery repeatedly fails and retries as a transient error in the Debugger, since a failed connection to the endpoint looks the same as a transient outage	Confirm Zerobus Ingest is available for your workspace's region in Databricks' ingestion availability documentation
Target table's columns no longer match the expected schema (`event_id`/`event_timestamp`/`event_type`/`payload`)	Delivery errors reported in the Debugger; not retried	Don't modify the table Twilio created. If the schema drifted, restore the original columns

If Twilio can't deliver events to your destination, Twilio sends an error notification to the Debugger (available in the United States (US1) region only). If Twilio continues to encounter delivery errors, Twilio notifies you every 20 minutes after the initial error.