Is there any possibility for events duplicated in destination?
Is there any possibility for events duplicated in destination? Due to server crash or power failure from segment side. While sending an event to destination if server is down due to some problems before updating the status, in that time how do you ensure that the events are sent or not
Niall here, one of the Success Engineers at Segment. That's a great question and something our engineers have thought long and hard on. We have this awesome blog post outlining our delivery system here: https://segment.com/blog/exactly-once-delivery/
It goes into a lot of detail on the engineering steps taken to ensure as close as possible to guaranteed once delivery. From a very high level we dedupe events by the following method:
For each message in our stream, we first check if we’ve seen that particular message, keyed by its id (which we assume to be unique). If we’ve seen a message before, discard it. If it’s new, we re-publish the message and commit the message atomically.
I'd recommend going over that blog post and if you have any further questions please do reach out to us via the contact form here https://segment.com/contact/
Thats excellent post. But it clarifies if the event is duplicated from the client side or consumer/worker crashes then the dedupe system will prevent from duplicates. My question is while delivering the event to destination, before getting the acknowledgement from the destination server if the server crashes then it will lead to duplicates right? How does segment handle these kind of errors
We record all responses from a destination, so if a destination returns a successful response we consider the message delivered, if we do not receive a successful response (invalid json, 500 errors, timeouts etc) we consider it not delivered.
To check your deliverability to a destination you can find this info in your destination settings
Creation of duplicates would not be the case, we have a mature retry policies in place (https://segment.com/docs/destinations/#retries) to ensure reliable delivery of events as expected.
If you have any specific questions about deliverability issues to a destination please ping our team directly (https://segment.com/contact) via a ticket and we can look into it.