You're in the right place! Segment documentation is now part of Twilio Docs. The content you are used to is still here—just in a new home with a refreshed look.

Data Warehouses

FREE ✓

TEAM ✓

BUSINESS ✓

ADDON x

What's a warehouse?

A warehouse is a central repository of data collected from one or more sources. This is what commonly comes to mind when you think about a relational database: structured data that fits neatly into rows and columns.

In Segment, a Warehouse is a special type of destination. Instead of streaming data to the destination all the time, we load data to them in bulk at regular intervals. When we load data, we insert and update events and objects, and automatically adjust their schema to fit the data you've sent to Segment.

When selecting and building a data warehouse, consider three questions:

What type of data will be collected?
How many data sources will there be?
How will the data be used?

Relational databases are useful when you know in advance the information you want to collect and how you want to link that information. This is usually the type of database used in the world of user analytics. For instance, a users table might be populated with the columns name, email_address, or plan_name.

Examples of data warehouses include Amazon Redshift, Google BigQuery, and Postgres.

When Segment loads data into your warehouse, each sync goes through two steps:

Ping: Segment servers connect to your warehouse. For Redshift warehouses, Segment also runs a query to determine how many slices a cluster has. Common reasons a sync might fail at this step include a blocked VPN or IP, a warehouse that isn't set to be publicly accessible, or an issue with user permissions or credentials.
Load: Segment de-duplicates the transformed data and loads it into your warehouse. If you have queries set up in your warehouse, they run after the data is loaded into your warehouse.

(information)