Question

Source Function Not Working as Expected

  • 23 April 2024
  • 3 replies
  • 73 views
P
Rank

Hello all!

We currently sync all of our Segment data into Snowflake but there is one event which has a format that upon Segments automatic un-nesting of properties into columns creates a column explosion and sync failure.

To overcome this we have been proposed creating the below flow.
 

  1. Segment Source via Function - this function is defined to intercept the malformed property and correct it. Creating an identical event besides that one event property being changed.
  2. Segment webhook Destination - this is a webhook (actions) destination which has as input our source which brings in the malformed event. This destination has a mapping configured to send to the webhook url defined by the Segment Source via Function.

All Testing is showing that the Source Function is behaving as expected where I can pass in an event with the malformed event and it corrects it. The webhook Destination when sending a test event shows that the event being sent was a success. The problem is that the Source, powered by the Source function, is not actually receiving the event.

Can someone point me in the right direction?

My current thinking is that I need some authentication headers but I’m not sure if that’s true or if it is how to define them.

Thanks in advance,

Paul Burns

3 replies

Chauntel Greaves
Rank
Userlevel 3

Hi @Paul Burns - this looks like something our support team should look into. I’ll raise it to them. 

P
Rank

Thanks @Chauntel Greaves! I am working with support as well, the more eyes the better though.

Chris Carrel
Rank
Userlevel 1
Badge

@Paul Burns to confirm you have an existing source and you want to forward a specific event to a function to fix the event payload and then save it into Snowflake.  There are a couple of ways you could tackle this:

  1. Update your Source function to handle all events and just pass through all events without modifying them except for the one event that needs to be changed.  Then swap your existing source to the Source function with your destinations connected.
  2. Create a Destination function that is attached to your source and accepts the event that needs modified and then send the event either back to the source or to a new source connected to Snowflake.  If you send it back to the original source, you will need a filter or method to prevent an infinite loop.
  3. Continue with the Webhook action, the URL in your action mapping needs to be the Webhook URL that is shown on the Overview tab of source that you created from the function.

On a side note, do you have Segment Protocols?  If you just want to remove the one trait from the event you can do that with a transformation, so no function needed.

Let me know if you have any questions.  I would be happy to show you examples of how I have configured this.

Chris

Reply


Terms & Conditions