Event delivery retries and event duplication
It can be difficult to achieve perfect stability to receive Event Streams continuously. When network infrastructure, third-party, or Twilio internal issues occur, Event Streams retries delivering an event for up to four hours.
Event Streams notifies you about event delivery issues. You'll get error logs in the Twilio Console by default. You'll receive a notification after the first error and then more notifications grouped at certain points: every 20 minutes, after every 10, 100, 1000 errors, and so on. You can also create custom Alarms for specific errors to trigger console indicators, email notifications, or webhooks.
Event Streams guarantees at-least-once delivery of events. When an event delivery fails, Twilio retries delivering the event. Multiple event delivery retries could result in receiving duplicated events. If you want to discard duplicated events, then you must deduplicate the events you receive.
When an event delivery fails, Twilio retries delivering the event several times. Twilio performs the retry with an exponential backoff with jitter, meaning that the interval between retries increases and includes some randomness. Because of this randomness, there's no fixed number of attempts. Twilio continues retrying delivery until it succeeds or four hours have passed, after which it stops retrying.
Events Streams might deliver events out of order. You can use each event's timestamp to sort events.
The following tables list potential errors and the resulting Event Streams behavior.
| Issue | Error code | Event Streams behavior |
|---|---|---|
| Error writing to Kinesis | 93101 | retry delivery |
| assume role error | 93102 | retry delivery |
| get shard count error | 93103 | retry delivery |
| get Kinesis Stream Name And Region error | 93104 | discard event |
| Issue | Error code | Event Streams behavior |
|---|---|---|
| customer returning 429 HTTP error | 93101 | retry delivery |
| customer returning other 400 HTTP error (not 429) | 93101 | discard event |
| customer returning any 500 HTTP error | 93101 | retry delivery |
The Segment sink doesn't send notifications when delivery errors occur.
Twilio might send the same event multiple times for several reasons, including but not limited to the following:
- Event delivery retries due to errors
- Incident remediation
- Internal temporary errors
Duplicated events are events that have the same id. The id is a SID like EZ00000000000000000000000000000000. If you require exactly-once delivery, then you need to implement deduplication in the service that receives the events.
Use the following general steps to deduplicate received events:
- Check if you previously received another event with the same event
id. - If you already received an event with the same
id, discard the incoming event. - Otherwise, store the event
id.
Info
If you use a webhook sink, then you should return a 200 success status code when you discard a duplicate event.
Otherwise, Event Streams attempts to retry delivery.