Menu

Expand
Rate this page:

Problems on delivery

What if my sink has problems?

We are aware that it is very difficult to achieve a perfect stability of your sink. Sometimes your infrastructure may have problems and you won't be able to receive the events. EventStreams is ready for that and incorporates several mechanisms to cope with these issues.

In the case of throttling or certain kind of errors, EventStreams will accumulate at least 5 million events for at least 24H.

EventStreams will notify you about any issue listed here using Twilio Debugger. Depending on your configuration you wil receive mailing events, lists in Twilio Console or both. You will receive a notification after the first error and then more grouped at certain points: every 20 minutes, every 10/100/1000.. error and so.

What does "throttle sink" means for EventStreams?

When a Webhook sink is unable to cope with the load created by Event Streams, they may return a THROTTLING error. This is a 429 HTTP error..

When the customer returs a throttling error or they are having issues on their side (i.e. webhook returning a 5XX error) EventStreams will reduce the rate of deliveries to this sink until it is acceptable (i.e. the resource returns 2XX). It will stay like that for a period of time, and then slowly grow again until get to full speed.

It is important to note that when a sink is throttled the SLA of 2 seconds delivery will not be applied. Also, if throttling results on a late delivery of more than 5 million events, then there is the possibility of losing information.

Full throttling flow:

1. Sink returns 429 or 5XX (webhook)

2. Sink gets labeled as "throttled"

3. Sink gets a max. delivery rate of 100* events / second

3.a: Sink returns another 429/5XX: go to 4

3.b: Sink does not return any other 429/5XX in the following 5* minutes: go to 5

4. Sink gets its max. delivery rate limited to a 99%* of the current one. Low boundary is zero (i.e. not receiving anything)

4.a: Sink returns another 429/5XX: go to 4

4.b: Sink does not return any other 429/5XX in the following 5* minutes: go to 5

5. Sink gets unlabeled as "throttled" and returns to the normal submission rate.

* note that these values are not fixed and may vary in the future

What does "retry delivery" means for EventStreams?

Under certain conditions (see tables below) the delivery of an event will be retried several times. This retry is performed with an exponential backoff with jitter, meaning that it will be retried several times increasingly separated from each other, and including some randomness in the time. This will be performed for at least 24H.

Note that the SLA of 2 seconds delivery only applies for the first delivery attempt of an event, not for the ones being retried. Also there is a significant chance of events being delivered out of order in the case of retries, so we strongly reccomend using the timestamp present on each one to sort them if needed.

Behavior for each sink

Kinesis Sink

Issue Error code EventStreams behavior
Error writing to Kinesis 93101 Retry delivery
assume role error 93102 Retry delivery
get shard count error 93103 Retry delivery
get Kinesis Stream Name And Region error 93104 discard event

Webhook Sink

Issue Error code EventStreams behavior
customer returning 429 HTTP error 93101 retry delivery & throttle sink
customer returning other 400 HTTP error (not 429) 93101 discard event
customer returning any 500 HTTP error 93101 retry delivery & throttle sink

Segment Sink

Currenty this sink does not notify the customer in case of issues

Rate this page:

Need some help?

We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd by visiting Twilio's Stack Overflow Collective or browsing the Twilio tag on Stack Overflow.

        
        
        

        Thank you for your feedback!

        Please select the reason(s) for your feedback. The additional information you provide helps us improve our documentation:

        Sending your feedback...
        🎉 Thank you for your feedback!
        Something went wrong. Please try again.

        Thanks for your feedback!

        Refer us and get $10 in 3 simple steps!

        Step 1

        Get link

        Get a free personal referral link here

        Step 2

        Give $10

        Your user signs up and upgrade using link

        Step 3

        Get $10

        1,250 free SMSes
        OR 1,000 free voice mins
        OR 12,000 chats
        OR more