Solved

eTimedout errors for Webhook Destination

  • 28 March 2024
  • 4 replies
  • 50 views
C
Rank
Badge

Hey all -- we have a few different platforms that we push data into that aren’t native Segment destinations. For these, we rely on the Webhook Actions destination to create these integrations. One issue we’re experiencing is a fairly quick time-out rate that’s preset within these destinations.

 

In short, if there’s no response from the endpoint we POST to within 5s, Segment auto-retries the request. What we’re seeing in actuality is that the endpoint is successfully receiving the request from Segment but due to some added platform latency, the response isn’t getting back to Segment in time. This causes multiple retries and duplicate data or actions to be taken by the destination platform. 

For the group, has anyone found a way to creatively solve for this retry rate? For the Segment Team end, is there the possibility to allow for configurable retry rates?  

icon

Best answer by Chauntel Greaves 6 April 2024, 02:35

View original

4 replies

Chris Carrel
Rank
Userlevel 1
Badge

@Corey Egan I don’t think you are going to be able to get around the default 5s time with a Webhook action.  There are a couple of ways to go about resolving this:
 

  1. Work with the platform you are sending the data to and see if they can optimize their endpoint to provide a response in a more reasonable time
  2. Segment events have a message ID so see if the endpoints have the ability to dedupe events based on this value
  3. Instead of using the Webhook, create a destination function so that you have more control over the request and response and when to “retry” events.
Chauntel Greaves
Rank
Userlevel 3

Hey @Corey Egan - I raised this to our support engineering team to take a look.

Chauntel Greaves
Rank
Userlevel 3

Closing the loop here with the ticket response from our Support Eng team: 

“For more context on this topic, each time a Destination Function is invoked, it has 5 seconds to complete, otherwise the entire function for that event will be retried. So, if the function takes longer than 5 seconds to complete, then it would be retried until it received a success response from the destination. Our retry feature sits after functions and the rates cannot be modified since it works the same for all destinations. Functions also have a timeout of 5 seconds, and I'm afraid this is a hard limit that cannot be modified.
 
Additionally, the most common cause for timeouts are multiple requests sent in sequentially, instead of in-parallel, or an external fetch call taking too long to return a response.
You can use the Tester Feature in the editor to determine the duration of your external requests.
 
I recommend checking this section of our docs explaining how to improve the speed of external requests: Improving Speed of External Requests.”
 
Destination functions FAQs

Does Segment retry failed function invocations?

Yes, Segment retries invocations that throw RetryError or Timeout errors (temporary errors only). Segment’s internal system retries failed functions API calls for four hours with a randomized exponential backoff after each attempt. This substantially improves delivery rates.

Retries work the same for both functions and cloud-mode destinations in Segment.

L
Rank
Badge

https://hookdeck.com is what you need. They’ll accept the webhook and pass it along to your destination.

Reply


Terms & Conditions