When we send JSON object type columns to Snowflake, Segment flattens them and create separate columns for each key. This exceeds the 5000 properties limit. What would be a good workaround for this?
We are thinking of sending data in Array format but it will require more effort in Snowflake to use such data.
Integrating JSON type columns to Snowflake instead of flattening
Userlevel 1
Hi
Thank you for the question! After asking around and trying to recreate this issue, it appears the 5000 property limit applies for Selective Sync. Are you seeing something like this in the Sync History UI?:
“We were unable to copy data to a temporary staging table because there are too many properties for collection [Snowflake URL] Please check for dynamically created properties The error was: 001431 (42601): SQL compilation error: column '$4097' does not exist.”
In theory, you could go into the Selective Sync UI and disable all properties that you don’t need and the sync will retry, the problem seems to be the 5000 Property limit in the UI itself.
I can also open up a support engineering ticket for you, if you don’t have one open already, to see if they have another workaround. Let me know!
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.