Objects Bulk API

The Batched Object Data the API accepts is a file of line separated objects, in JSON form, compressed using Gzip. The maximum size of a single object is 400 KB and maximum uncompressed size of a file 512 MB.

Example objects:

1
{"id":"1","collection":"users","properties":{"first_name":"John","last_name":"Smith"}}
2
{"id":"2","collection":"users","properties":{"first_name":"Jane","last_name":"Doe"}}

Authenticate to the Objects Bulk API by sending your project's Write Key along with a request. Authentication uses HTTP Basic Auth, which involves a username:password that is base64 encoded and prepended with the string Basic .

In practice that means taking a Segment source Write Key encoding it with base64. For example, echo "abc123" | base64 - becomes 'YWJjMTIzCg==' and this is passed in the authorization header like so: 'Authorization: Basic YWJjMTIzCg=='.

set up an HTTP API source type in Segment. You will use this source write key for authenticating with the Objects Bulk API.

The API imposes some rate limits including:

• 512 MB maximum uncompressed file upload size
• 400 KB maximum object size
• 20 simultaneous requests from the same IP
• 20 simultaneous requests per Write Key

Start indicates the beginning of a session to upload/publish data. It Returns a unique ID to be used in subsequent endpoints denoted by <id>. Sessions are short lived, see Keep Alive for details on keeping your session alive.

Example Start request:

POST https://objects-bulk-api.segmentapis.com/v0/start

Example Start response

1
{
2
  "sync_id":"hjWdsur28j49LCQ"
3
}

Possible HTTP responses include:

• 201 Created
• 400 Bad Request
• 500 Internal Server Error

Upload is used to publish the file(s) of batched object data to be processed. It returns a unique ID associated with the published file for future use.

Example Upload request:

1
POST https://objects-bulk-api.segmentapis.com/v0/upload/<id>
2
Content-Type: binary/octet-stream
3
Content-Encoding: gzip

Example Upload response:

1
{
2
  "part_id":"dkf539GFfmj50fnm"
3
}

Possible HTTP responses include:

• 201 Created
• 400 Bad Request
• 413 Entity Too Large
• 500 Internal Server Error

Finish is used to indicate that this session is complete and no more files will be uploaded. It accepts an error message which indicates if the session was successful.

Example Finish request:

1
POST https://objects-bulk-api.segmentapis.com/v0/finish/<id>
2
Content-Type: application/json

1
{
2
  "error":"Blank if everything was successful"
3
}

Possible HTTP responses include:

• 202 Accepted
• 400 Bad Request
• 500 Internal Server Error

Keep Alive is used to extend your sessions lifetime if there are long gaps between API calls. The default session timeout is 10 minutes.

Example Keep Alive request:

POST https://objects-bulk-api.segmentapis.com/v0/keep-alive/<id>

Possible HTTP responses include:

• 200 Ok
• 500 Internal Server Error

Segment recommends that you use snake case when naming any object properties.

1
{
2
  "properties": {
3
    "name": "Charming Beach Room Facing Ocean",
4
    "location": "Lihue, HI",
5
    "review_count": 47
6
  }
7
}

You should use plural collection names wherever possible. The collection name should describe the group of one or many objects within the collection.

1
"collection": "rooms"
2
"collection": "products"
3
"collection": "reviews"

Segment de-dupes objects with the same ID and merges properties. By sending in partial objects with the same object ID, Segment merges the properties and you can query the latest data in your data warehouse.

For example, if you make the following requests with the following payloads:

1
{"id": "2561341","collection": "rooms","properties": {"name": "Charming Beach Room Facing Ocean","location": "Lihue, HI","review_count": 47}}
2
{"id": "2561341","collection": "rooms","properties": {"name": "Charming Beach Room Facing Ocean","location": "Lihue, HI","review_count": 48}}

When you query your warehouse you will see the following:

select id, name, location, review_count from airbnb.rooms

 '261341' | 'Charming Beach Room Facing Ocean' | 'Lihue, HI' | 48

If the program that collects your data runs a predefines schedule, then use Objects Bulk API. If the data being collected is streaming all of the time then the Objects API is more suitable.

Contact Segment if you would like this feature.