A complete, modern analytics solution should harness both instrumented events and unstructured data
Imagine walking down the street one night, and you encounter someone searching for their keys under a streetlight. You offer to help, but neither of you can find the keys.
When you ask, “Are you sure you lost them here?”, they reply, “No, I lost them in the park.”
“Then why are you looking here?”
“Because this is where the light is.”
This is known as the Streetlight Effect, and is a surprisingly common observational bias. We seek answers from the information we can easily access, rather than where answers can actually be found. In product analytics, this manifests in the debate over the merits of manual instrumentation of data versus automatic data capture (we call it “autocapture,” while some refer to it as “auto-track”).
But this whole argument is based on a false dichotomy. If relying solely on manual instrumentation is like looking for your keys under the streetlight, then relying solely on autocapture is explicitly avoiding the streetlight. In the quest to improve your customers’ digital experiences, you should take advantage of all the data and insights at your disposal. Fullstory has built a platform that lets you leverage the strengths of both paradigms, and we believe that this approach is the way forward for digital experience and product analytics.
All digital experience data is valuable
Before we can illuminate why the answer is both, not either, we need to break down digital experience (DX) data into a few categories that product and digital teams use in different ways.
Core Events and Variables: The events and variables that support consistent KPIs. These are part of a product plan, so they’re largely known in advance and are valid and valuable over long periods of time. These variables include values often found in a page’s “data layer,” such as customer information, traffic source, cart value, and so forth. KPIs that are defined by this small subset of events include things like daily active users (DAU), conversion, and bounce rates or even mobile app logins.
Transient Events: Some metrics come and go as your product changes. A product manager will add and remove these as features, and the questions you have about them come and go. Agile product organizations build, ship, and iterate so often that these transient metrics can lose meaning quickly. They include things like new feature usage (until that feature is no longer new), interactions with specific CTAs or content, or even engagement with seasonal or temporal shopping experiences.
The Long Tail: These are all the other things that occur between your product and customers that you couldn’t manually instrument but hold an untold quantity of user intent and interaction insights. Think about the plethora of navigation flows, form and UI component interactions, errors, performance and network logs, and the very rich detail of full session recreations replete with scrolling, clicks, highlights, and observable issues like cringe-worthy workflow loops or the much-feared Rage Click™.
Capturing ALL the DX intelligence requires a “both,” not “either,” approach
A small subset of Core and Transient Events is typically part of any good analytics implementation and is usually instrumented manually. Basic usage metrics of new features being launched may also get tagged manually, though they are more subject to mistakes, staleness, or drift over time that render them useless within a few months.
But most events are never gathered at all. And these are often the ones that provide the most important context to your top-line metrics and can uncover unexpected “AHA” moments.
We can map these three types of data to Fullstory’s approach, which includes both instrumented events and unstructured DX data. The image below represents the totality of DX data. Instrumented events are composed primarily of core events and variables and a handful of transient events. The sheer volume of long-tail data that resides in the unstructured DX realm is massive and represents extremely valuable signals you could never hope to instrument.
A complete, modern analytics solution should harness all of this information, combining all of your manually instrumented events and variables with unstructured experience data—effectively providing daylight so you’re not limited to streetlights to find your keys. Your product analytics solution should be advanced enough to help you spot problems and derive insights, wherever they are—not limit you to the ones that happen to be under the streetlight.
You can’t uncover unknown unknowns without ALL the data
Not long ago, our team engaged with a prospective ecommerce customer. They already had a solution on board that they’d used to carefully tag their site and checkout flow. As part of their due diligence, they dropped Fullstory’s single line of code on their site and let both solutions run for about a month.
As luck (or rather bad luck) would have it, they ran a big new promotion during that time. Their checkout flow suffered under the weight of shopper demand. Fullstory was able to quickly uncover a highly unusual error that the other solution didn’t pick up—because it wasn’t looking for it. This was the kind of obscure error—an unknown unknown—that no developer or PM would ever think to instrument.
What took the retailer a matter of hours to identify and fix with Fullstory would have taken days and cost millions in lost revenue with an instrumented approach – or worse, might have never been uncovered.
We didn't just help the brand and win their business; we changed some minds about the richness of insights to be found in the long tail of the data.
Maintaining data integrity as products evolve is easier with autocapture
If you’re building events and metrics on top of a site or application that’s constantly changing, how can you be sure they won’t break when your product team ships an update? The short answer is that they can break, but if you think you have to choose between “instrumented events that cover everything and always work” and “unreliable chaos,” you’re asking the wrong question.
All product metrics can—and do—break, no matter how they’re implemented. Manually instrumented events break for all sorts of reasons: products evolve, people forget to update things, definitions change. The important question is “What happens when they break?”
With manual instrumentation, you start losing data the moment an event breaks. It’s even hard to know whether an event is broken; the best you can do is inspect the code when it looks like your data might be wrong. That could result in a lot of engineers hunting and pecking based on a hunch that something “is not right.”
Autocaptured events can also break, for similar reasons. But, it’s a lot easier to know whether and why an event broke because you can replay sessions to see what users actually did and quickly fix the issue. Better yet, when an event that is being autocaptured breaks in Fullstory, your metrics can be updated retroactively, so you don’t lose anything.
For a deeper look at all things autocapture, read the ultimate guide here.
Comprehensive data capture powers a privacy-first approach
How is it possible to capture so much data safely, without compromising your users’ privacy or security? The answer lies in the comprehensive nature of Fullstory’s autocapture approach.
For the bulk of use cases—analytics, usability, support, error detection—you actually don’t need personal user data. The structure and metadata—like page structure, errors, performance data, mouse position, scrolling—are required, but personal information or even the specific images or icons on a page aren’t necessary to get value from the data. Fullstory can capture and recreate entire sessions and extract structured events without capturing the content presented to users or any data they share with your company.
We call this approach “Private by Default” (PbD). PbD allows you to selectively capture only those parts of the UI and other elements that you know to be safe, while still getting the data you need to understand user behavior and troubleshoot problems. Solving this efficiently is complex and requires powerful configuration tools that let you easily identify the areas you want to capture, without having to drill into every little detail of your product. We believe it’s the only safe way to address this problem.
To the future, with autocapture
Does this mean that product analytics is completely solved? Is there daylight everywhere to find those keys? Realistically, we know there’s always more to be done to make product analytics higher-leverage, more insightful, and more reliable. We continue to learn from our thousands of customers and are expanding Fullstory to power a better understanding of their users.
When it comes to privacy and security, we’re making deep R&D investments to help customers automatically detect common types of personal information. This method will serve as a back-stop against accidentally capturing unintended data, be it instrumented or autocaptured. This will add another layer of safety, allowing you to rest easy knowing your users’ data is safe.
We’re also building machine learning models that use all the data in captured sessions to understand every aspect of your product—every page, screen, button, form, obscure error box, and A/B test variation. This will allow us to intelligently suggest new events, learn new elements, and heal event definitions that suffer from drift over time. These are precisely the unknown unknowns that could result in major cost savings and opportunities to delight customers. Far from being a burden—or a privacy concern—capturing and analyzing more data is the solution to the problem of complex and rapidly changing products.
A hybrid approach of both manual instrumentation and autocapture is the only path to uncovering the product and DX insights that help businesses thrive—and Fullstory’s mission is to help you make sense of that data to create more perfect digital experiences for users everywhere. To the future!
Want to learn more? See Fullstory's unique approach to data collection in this short video
