Problems on delivery

What if my sink has problems?

We are aware that it is very difficult to achieve a perfect stability of your sink. Sometimes your infrastructure may have problems and you won't be able to receive the events. EventStreams is ready for that and incorporates several mechanisms to cope with these issues.

In the case of throttling or certain kind of errors, EventStreams will accumulate at least 5 million events for at least 24H.

EventStreams will notify you about any issue listed here using Twilio Debugger. Depending on your configuration you will receive mailing events, lists in Twilio Console or both. You will receive a notification after the first error and then more grouped at certain points: every 20 minutes, every 10/100/1000.. error and so.

What does "throttle sink" means for EventStreams?

When a Webhook sink is unable to cope with the load created by Event Streams, they may return a THROTTLING error. This is a 429 HTTP error..

When the customer returs a throttling error or they are having issues on their side (i.e. webhook returning a 5XX error) EventStreams will reduce the rate of deliveries to this sink until it is acceptable (i.e. the resource returns 2XX). It will stay like that for a period of time, and then slowly grow again until get to full speed.

It is important to note that when a sink is throttled the SLA of 2 seconds delivery will not be applied. Also, if throttling results on a late delivery of more than 5 million events, then there is the possibility of losing information.

Full throttling flow:

1. Sink returns 429 or 5XX (webhook)

2. Sink gets labeled as "throttled"

3. Sink gets a max. delivery rate of 100* events / second

3.a: Sink returns another 429/5XX: go to 4

3.b: Sink does not return any other 429/5XX in the following 5* minutes: go to 5

4. Sink gets its max. delivery rate limited to a 99%* of the current one. Low boundary is zero (i.e. not receiving anything)

4.a: Sink returns another 429/5XX: go to 4

4.b: Sink does not return any other 429/5XX in the following 5* minutes: go to 5

5. Sink gets unlabeled as "throttled" and returns to the normal submission rate.

* note that these values are not fixed and may vary in the future

What does "retry delivery" means for EventStreams?

Under certain conditions (see tables below) the delivery of an event will be retried several times. This retry is performed with an exponential backoff with jitter, meaning that it will be retried several times increasingly separated from each other, and including some randomness in the time. This will be performed for at least 24H.

Note that the SLA of 2 seconds delivery only applies for the first delivery attempt of an event, not for the ones being retried. Also there is a significant chance of events being delivered out of order in the case of retries, so we strongly reccomend using the timestamp present on each one to sort them if needed.

Behavior for each sink

Kinesis Sink

Webhook Sink

Segment Sink

Currenty this sink does not notify the customer in case of issues