--- "@context": https://schema.org "@type": TechArticle "@id": https://www.twilio.com/docs/data-destinations/destination/databricks#article headline: Databricks destination description: Learn how to send event data to Databricks Delta tables for lakehouse analytics. url: https://www.twilio.com/docs/data-destinations/destination/databricks inLanguage: en dateModified: 2026-11-04T17:24:28.000Z author: "@type": Organization name: Twilio Developer Education Team publisher: "@type": Organization name: Twilio --- # Databricks destination > \[!IMPORTANT] > > Data Destinations is currently available as a Private Beta product and the information contained in this document is subject to change. Data Destinations is not a HIPAA Eligible Service or PCI compliant and should not be enabled in workflows that are subject to HIPAA or PCI. To learn more, see [Private beta legal information](/docs/data-destinations#private-beta-legal-information). [Databricks](https://www.databricks.com/) is a data lakehouse platform for analytics, ML, and data warehousing. Sending events to Databricks lets you stream communications data directly into a Delta table for low-latency SQL analytics and downstream ML. Twilio delivers events using [Zerobus Ingest](https://docs.databricks.com/aws/en/ingestion/zerobus-overview), Databricks' push-based streaming service that writes records directly into a Unity Catalog Delta table with no intermediate broker. Twilio streams each event straight into your table over a gRPC connection, and the event is typically queryable within seconds. Check Databricks' [ingestion availability](https://www.databricks.com/product/pricing/lakeflow-connect) documentation to confirm Zerobus Ingest is available in your workspace's region before setting up this destination. ### Destination overview * **Category**: Warehouse * **Release status**: Beta * **Authentication**: OAuth (client ID and secret) * **Regional availability**: US1 ## Join the Data Destinations private beta [Sign up](https://airtable.com/appaJfyV6LiIFMxuP/pagIaOAtKsQVxTfdW/form) Get early access to native destinations for data warehouses, observability tools, and event pipelines. ## Eligible events You can send all [event types](/docs/events/event-types-overview) supported by Event Streams to Databricks. This destination has no user-identifier restriction. ## Get started ### Prerequisites Before you set up a Databricks destination, make sure you have: * A Databricks workspace on Amazon Web Services, Azure, or Google Cloud. * A Databricks **service principal** with an OAuth **Client ID** and **Client Secret** (found in your Databricks account under Service Principals). * An existing Unity Catalog **catalog** and **schema** that the service principal has `USE CATALOG`, `USE SCHEMA`, `CREATE TABLE`, `MODIFY`, and `SELECT` privileges on. `CREATE TABLE` is needed for initial provisioning of the destination table (`twilio_events_`), and `MODIFY`/`SELECT` are needed for ongoing Zerobus Ingest writes and reads. The catalog and schema themselves must already exist. Twilio creates only the table. * A SQL warehouse for the service principal to use when Twilio creates the destination table. ## Create the destination > \[!NOTE] > > See [Set up a destination](/docs/data-destinations/destination-setup) for the full UI and Destinations API setup flow common to every destination type. ### Twilio Console To create a Databricks destination in the Twilio Console: 1. From your Databricks account, gather the settings listed in [Destination settings](#destination-settings). 2. In the Twilio Console, go to **Data** > **Data Destinations** > **Destinations** > **Add destination subscription**. 3. Complete the **Add destination subscription** wizard. ### Destinations API To create a Databricks destination through the Destinations API, send a `POST` request to the Destinations endpoint and set `destinationType` to `databricks`. For authentication and the other request body fields, see [Create a destination](/docs/data-destinations/api-quickstart#create-a-destination). ```text title="Create a destination" POST "https://destinations.twilio.com/v1/ControlPlane/Destinations" ``` Set the `config` object in the request body as follows: | Name | Type | Description | | ------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------- | | `catalog`
(**Required**) | string | Unity Catalog catalog that holds the target tables. | | `credentials.credentialType`
(**Required**) | string | Set to `CUSTOM`. | | `credentials.clientId`
(**Required**) | string | The application ID (UUID) of your OAuth machine-to-machine service principal. | | `credentials.clientSecret`
(**Required**) | string | The OAuth secret generated for the service principal. | | `schema`
(**Required**) | string | Unity Catalog schema (inside catalog) that holds the target tables. | | `warehouseId`
(**Required**) | string | The SQL warehouse that creates the per-subscription Delta table when you save a subscription. | | `workspaceUrl`
(**Required**) | string | Your Databricks workspace URL, for example `https://dbc-abc123.cloud.databricks.com`. | | `zerobusEndpoint`
(**Required**) | string | The Zerobus gRPC ingest endpoint host, in the format `WORKSPACE_ID.zerobus.REGION.cloud.databricks.com`. | The following example shows a request body. Replace the uppercase placeholder values with your own. ```json title="Example request body" { "destinationType": "databricks", "name": "My Databricks destination", "config": { "catalog": "CATALOG", "credentials": { "credentialType": "CUSTOM", "clientId": "CLIENT_ID", "clientSecret": "CLIENT_SECRET" }, "schema": "SCHEMA", "warehouseId": "WAREHOUSE_ID", "workspaceUrl": "WORKSPACE_URL", "zerobusEndpoint": "ZEROBUS_ENDPOINT" } } ``` ## Destination settings Use `databricks` as the `destinationType`. | Field | Description | Required | Default or choices | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------ | | Client ID | OAuth service-principal Application ID (UUID). Sensitive, stored encrypted. | Yes | None | | Client Secret | OAuth secret generated for the service principal. Sensitive, stored encrypted. | Yes | None | | Workspace URL | Your Databricks workspace URL. For example, `https://dbc-abc123.cloud.databricks.com`. | Yes | None | | Zerobus Endpoint | The Zerobus gRPC ingest endpoint host for your workspace's region, in the form `.zerobus..cloud.databricks.com`. | Yes | None | | Catalog | The Unity Catalog catalog that holds the target table. You can't change this after creating the destination. | Yes | None | | Schema | The Unity Catalog schema (inside Catalog) that holds the target table. You can't change this after creating the destination. | Yes | None | | SQL Warehouse ID | The SQL warehouse Twilio uses to create the destination table for you. | Yes | None | Twilio authenticates to Databricks by using OAuth 2.0 machine-to-machine (client credentials) with your service principal's **Client ID** and **Client Secret**. Twilio exchanges these for a short-lived access token and sends it as a bearer token on every request. Twilio never sends your Client Secret itself to Zerobus. The **SQL Warehouse ID** and **Zerobus Endpoint** serve two different purposes. Twilio uses the SQL warehouse once when accessing the SQL Statement Execution API, to create the destination table the first time you save a subscription. Twilio streams every event after that directly into the table over Zerobus Ingest. The SQL warehouse isn't involved in ongoing delivery. The first time you save a subscription, Twilio creates the destination table `twilio_events_` in your configured catalog and schema. ## Default mappings | Field | Databricks column | | ------------- | ----------------- | | Event ID | `event_id` | | Event type | `event_type` | | Event time | `event_timestamp` | | Event payload | `payload` | ## Verification and troubleshooting When you save a Databricks destination, Twilio validates your Client ID and Client Secret by requesting an OAuth token from your workspace. If validation fails, check that: * Your **Client ID** and **Client Secret** are correct and haven't been revoked in Databricks. * Your **Workspace URL** is correct and reachable. This validation confirms your credentials only. It doesn't confirm that the service principal has the required Unity Catalog grants or that your catalog and schema exist. If your principal is missing the required Unity Catalog grants or your catalog and schema don't exist, table creation or event delivery might fail after you save the destination. If events aren't arriving as expected, check your service principal's grants on the target catalog and schema. Twilio doesn't currently offer a way to trigger a test event for this destination from either the Console or the Twilio command line interface (CLI). To confirm delivery is working, wait for a real event of a subscribed type to fire. ### Common errors | Cause | What you'll see | Fix | | ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Invalid or revoked Client ID / Client Secret | Validation fails with an auth error | Re-check the credentials in Databricks and update the destination | | Service principal missing Unity Catalog grants | Table creation or delivery fails after the destination validates | Grant `USE CATALOG`, `USE SCHEMA`, and `CREATE TABLE`/`MODIFY`/`SELECT` to the service principal on your catalog and schema | | Wrong region in Zerobus Endpoint | Delivery errors reported in the [Debugger](/docs/usage/troubleshooting/debugging-your-application) | Confirm the endpoint's region matches your workspace's region | | Zerobus Ingest isn't available in your workspace's region | Delivery repeatedly fails and retries as a transient error in the [Debugger](/docs/usage/troubleshooting/debugging-your-application), since a failed connection to the endpoint looks the same as a transient outage | Confirm Zerobus Ingest is available for your workspace's region in Databricks' [ingestion availability](https://www.databricks.com/product/pricing/lakeflow-connect) documentation | | Target table's columns no longer match the expected schema (`event_id`/`event_timestamp`/`event_type`/`payload`) | Delivery errors reported in the [Debugger](/docs/usage/troubleshooting/debugging-your-application); not retried | Don't modify the table Twilio created. If the schema drifted, restore the original columns | If Twilio can't deliver events to your destination, Twilio sends an error notification to the [Debugger](/docs/usage/troubleshooting/debugging-your-application) (available in the United States (US1) region only). If Twilio continues to encounter delivery errors, Twilio notifies you every 20 minutes after the initial error. ## Additional resources * [Destinations overview](/docs/data-destinations/destinations-overview) * [Set up a destination](/docs/data-destinations/destination-setup) * [Event types reference](/docs/events/event-types-overview)