0

Is there any possibility for events duplicated in destination?

Is there any possibility for events duplicated in destination? Due to server crash or power failure from segment side. While sending an event to destination if server is down due to some problems before updating the status, in that time how do you ensure that the events are sent or not

5replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Hey there,

    Niall here, one of the Success Engineers at Segment. That's a great question and something our engineers have thought long and hard on. We have this awesome blog post outlining our delivery system here: https://segment.com/blog/exactly-once-delivery/

    It goes into a lot of detail on the engineering steps taken to ensure as close as possible to guaranteed once delivery. From a very high level we dedupe events by the following method:

    For each message in our stream, we first check if we’ve seen that particular message, keyed by its id (which we assume to be unique). If we’ve seen a message before, discard it. If it’s new, we re-publish the message and commit the message atomically. 


    I'd recommend going over that blog post and if you have any further questions please do reach out to us via the contact form here https://segment.com/contact/

    Thanks!

    Reply Upvote
  • Thats excellent post. But it clarifies if the event is duplicated from the client side or consumer/worker crashes then the dedupe system will prevent from duplicates. My question is while delivering the event to destination, before getting the acknowledgement from the destination server if the server crashes then it will lead to duplicates right? How does segment handle these kind of errors

    Reply Upvote
  • We record all responses from a destination, so if a destination returns a successful response we consider the message delivered, if we do not receive a successful response (invalid json, 500 errors, timeouts etc) we consider it not delivered.

    To check your deliverability to a destination you can find this info in your destination settings

    https://segment.com/docs/guides/destinations/how-do-i-check-if-data-is-successfully-being-delivered-to-my-destination/

    Reply Upvote
  • Yes i know that, but my question is different, while sending the data to destination if the processing unit is down then it may create duplicates right

    Reply Upvote
  • Creation of duplicates would not be the case, we have a mature retry policies in place (https://segment.com/docs/destinations/#retries) to ensure reliable delivery of events as expected. 

    If you have any specific questions about deliverability issues to a destination please ping our team directly (https://segment.com/contact) via a ticket and we can look into it.

    Reply Upvote