3

Automated testing / QA tools

Curious if anyone has had any success using (automated) testing tools to monitor & QA event tracking.

We're using Segment's analytics.js and node.js libraries to track frontend and backend events, respectively, and use the Warehouse offering to store all the data in Redshift. Lots of great ways to dig into the event data once it's captured, but I haven't found a great way to ensure we're capturing everything...

When we see an issue with tracking, we typically have to turn to log files and a deep dive - I'm eager to find a proactive solution that tests all events are firing as intended.

Many thanks!

8replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Hmmm we have performance monitoring tools that are on our platform. You can check them out on our integrations page (scroll to bottom, then select "Monitoring" category). But those are usually for monitoring errors in your frontend/backend.

    What kind of "issue with tracking" are you referring to? Data discrepancy between tools vs your DB counts?

    Reply Upvote
  • Thanks Han. The problem I'm trying to solve is confidence in our tracking and general data hygiene: how can we be sure we're actually tracking all pageviews, clicks, modal pops, submit events, etc...  Imagine if we had a bot that went through our flow 100 times - I'd expect to see each unique event along the funnel to fire exactly 100 times... however I'm not confident that is actually the case today for a variety of reasons.

    With regards to "issue with tracking" I'm referring to a difficulty in reconciling counts of front-end clicks and back-end objects created. For example, I see cases where the CTA that leads to a back end event (e.g. 'User Click Buy' --> 'Order Created') has *lower* counts than the subsequent event... Additionally - sometimes the track() calls will either 1) send data to an incorrect field or completely skip a specific field. This can happen for various reasons, everywhere from implementation errors on our part, to page load time, to issues in data processing or Segment Warehouse ETL.

    Do others see this issue? If so, how do they handle it? If not - why not? :)

    Reply Upvote
  • Akhill Chopra  Have you got any information about the automation? Even I am also looking for the same!

    Any information is much appreciated here.

     

    Thanks!

    Reply Upvote
  • Akhill Chopra Muthu Thangavelsamicheck out http://observepoint.com/

    Reply Upvote
  • The same question here. I see that there are lots of front-end scanning tools, but those won't be enough. I would need a tool that could also check the back-end events and data integrations. E.g. if the user submits the web form, then the test should validate that the event was fired, GA recorded the conversion and the new record was created in API integrated CRM (with all the required acquisition/UTM data attached).

    Reply Upvote 1
  • Han Kim As below Akhill Chopra states a good scenario that is the case I would like to solve for also.

    Thanks Han. The problem I'm trying to solve is confidence in our tracking and general data hygiene: how can we be sure we're actually tracking all pageviews, clicks, modal pops, submit events, etc...  Imagine if we had a bot that went through our flow 100 times - I'd expect to see each unique event along the funnel to fire exactly 100 times... however I'm not confident that is actually the case today for a variety of reasons.

    With regards to "issue with tracking" I'm referring to a difficulty in reconciling counts of front-end clicks and back-end objects created. For example, I see cases where the CTA that leads to a back end event (e.g. 'User Click Buy' --> 'Order Created') has *lower* counts than the subsequent event... Additionally - sometimes the track() calls will either 1) send data to an incorrect field or completely skip a specific field. This can happen for various reasons, everywhere from implementation errors on our part, to page load time, to issues in data processing or Segment Warehouse ETL.

    Do others see this issue? If so, how do they handle it? If not - why not? :)

    Can you help with this scenario and provide feedback, advice on an efficient method/solution for ongoing testing and data validation. At present it seems I am limited to having to manually run through the entire funnel in UAT site and have segment debugger running to validate the right events, properties and or traits are firing, being collected.

    Simon mentions http://observepoint.com/ as an option.

    I think it would be great to know how your implementation engineer's, success managers work with premium support clients to conduct or advise on full funnel testing to be confident with data integrity when pushing new releases or adding, updating events.

    Thanks

    Matt

    Reply Upvote
  •  Akhill Chopra did you find a solution? 

    cc Han Kim

    Reply Upvote
  • Jeff Benton We release a product called Protocols that will help companies install data governance. We will now let you define a "tracking plan" that defines what events + props you expect and then Protocols will enforce the tracking plans' schema and alert you in realtime if any bad data is coming into our pipeline. 

     

    You can read more about this here: segment.com/protocols

     

    The folks here that are having data quality issues or data discrepancy issues should move any events that are mission critical (ie. Order Completed) to server side tracking. Client side tracking has inherent risks due to adblockers, browser redirects and such. 

     

    Protocols will help you avoid things like duplicate events, inconsistent event taxonomy across teams, and seeing unexpected / incorrect values or fields and yes — most of these issues are caused by implementation errors, lack of alignment between consumers / stakeholders of the data with the engineers who are tagging the sources, etc much more. These issues are being solved by leveraging our Protocols product. (Caveat is that this is a business tier addon product and is not available to team plans. Please email me at han@segment.com if you want to discuss going enterprise route to leverage the data governance layer of the Segment platform.

     

    Data discrepancy issues are more related to _how_ you are tracking data such as client side versus server side. This should be done on a case by case basis as some events you cannot fire from server side or need to fire from client side (ie. mapping a track event to fire a client side ad pixel). 

    Reply Upvote