The qualitative + quantitative digital experience flywheel

Quantitatively analyze your qualitative digital experience data to create a virtuous cycle for digital product improvements.

Understanding how people interact with your website, app, or digital product is hard. You can have all the user data in the world but no idea what to do with it, no process to manage it, and no way to filter the signal from the noise.

Where do you even start? And once you begin, where do you go?

“Data saturation is everywhere. We’ve often had the belief that more is better; however, that actually isn’t true in the case of data. The rapid rise in our ability to collect data hasn’t been matched by our ability to support, filter and manage the data. [We have] too much data with not enough structure in place to manage the data and not enough meaningful application.”
— Sara Spivey, CMO at Bazaarvoice (Forbes)

What you need is a systemized approach—and the right tools.

When it comes to knowing, traditionally there have been two approaches you can take: qualitative and quantitative.

What is quantitative analysis?

Quantitative analysis of digital experience data puts everything in terms of numbers. The number of users clicking a button, bounce rates, time on site, and more. Quantitative research is a clean way to crunch numbers and understand what’s going on at a high level. 

The drawback to quantitative data is that much can be lost in translation. Important nuances missed and in that absence of information like reviewing user sessions, the quantitative can play tricks on our pattern-recognizing brains.

What is qualitative analysis?

To really understand what happens on your website, app, or web application, you also need to understand why users took certain actions. And that's where the second approach comes in—the qualitative.

Qualitative analysis focuses on the subjective qualities of your users—the nuanced actions that drive the numbers. It's descriptive research. The qualitative approach is subjective, too. The user paused here. Rage Clicked there. They seem confused with this UI—what's going on? Qualitative data is messy. But it is rich in insights.

The major problem with qualitative research methods is that they occur at the individual level. Qualitatively researching every user who visits your site is impossible. You could never do it, nor would it be useful.

The best analyses combine quantitative and qualitative research to create a flywheel of continuous product improvement

So what do you do? Use both.

Collectively, quantitative and qualitative analysis is the backbone of Digital Experience Intelligence.

The best analyses combine quantitative and qualitative research to create a flywheel of continuous product improvement. You use quantitative information to focus attention on the biggest problems (and opportunities) and then "zoom in" to the level of the user through qualitative research to round out your understanding—and solve the problem.

This one-two research method spins out insights you can turn into actions. It's a quantitative-qualitative flywheel that can be used for continuous improvement of your website or app.

Here's the 3-step process:

1. Quantitative data alerts you to problems

The most basic web analytics tools available are quantitative—and free. It comes as no surprise that one of the first things web site owners do on launching a new site is install Google Analytics.

Traditional quantitative analytics tools like Google Analytics do a reliable job providing high-level metrics—such as user info, page visits, events, conversion rates, bounce rates, and time on site. Most importantly, they allow you to see the status of different aspects of your site and help you focus on problems or opportunities in aggregate.

With a functional quantitative analytics tool up and running, you're ready to identify issues and opportunities for improvement. This is a great place to put in motion a quantitative-qualitative flywheel—though as you'll see in this article, once you have the flywheel going, it's not the only place to begin.

When you only have a quantitative tool tracking your customer data, it is easy to be dazzled by the illusion of insights. Come at your newfound quantitative data powers with an objective to avoid becoming overwhelmed.

Consider these changes in quantitative metrics that left team's struggling to understand the why behind the quantitative what:

• An increase in customer churn seen by Wistia couldn't be explained or reproduced. Why were customers churning?

• A drop in conversions like the car insurance company Metromile saw when users started quote requests on their site but then left them unfinished. Why weren’t users completing the form?

• Decreased usage of certain features, like the customer service software company Kayako saw on analyzing feature adoption. Why weren't customers trying out new features?

• Increased page bounces, seen by the translation service SpanishDict on on their homepage. What was breaking at the moment of first engagement?

Remember: behind every quantitative metric is a real person trying to get some job done. You have to understand what your users and customers are trying to accomplish to smooth out the digital experience—and quantitative information can’t show you the nuance.

To figure out what jobs they want to get done, you need to build out the next step in the flywheel.

2. Qualitative data gives you the why

Having used quantitative analysis to identify problems that require additional analysis, you need a way to "zoom in" on individual user behavior. That's why you need a qualitative analytics tool alongside a quantitative one: for total digital experience intelligence (DXI).

Qualitative data can provide you with a ton of nuanced information about individual user behaviors. It's the most dense level of data. It's the kind of data you get with open-ended feedback—and also what you get through session replay. Using a qualitative analytics tool, you can see how individual users interact with your product.

So how does it work?

How do you go from quantitative data to qualitative research? Using a digital experience intelligence platform like Fullstory, you can search the entire catalog of user behaviors on your site for just those that would affect whatever quantitative data you're researching.

Continuing from our company examples above, qualitative data was used to uncover the why behind the change in quantitative data:

• Rage clicking indicated to Wistia that much of their customer churn was caused by confusion about how to use the product.

• Confused site navigation on Metromile's online dashboard showed the company that users didn't understand the value of their pay-per-mile model.

• Observing that most users skipped onboarding shed a light on why Kayako users didn't know about certain features or couldn't figure out how to use them.

• Seeing the site layout from the user's perspective helped SpanishDict see that users couldn't figure out how to get started because a banner ad was blocking the search bar.

In each of the above cases, qualitative research (completed using Fullstory) provided the additional insights necessary to make sense of the problems identified through other quantitative tools. Collectively, that’s digital experience intelligence.

Having used qualitative research to understand why you're seeing changes in quantitative data, it's time for action.

3. Experiment and check for improvements

Armed with a complete view of the user from the individual to the aggregate, you now have the insight you need to do something, and that something is to make edits and run experiments. This is the last step in the quantitative-qualitative flywheel.

Just what edits are required will vary greatly depending on the problem you're solving. Make the edits, run the experiments (A/B testing is helpful here), and then measure your results. You're back at the top of the flywheel, and on and on we go.

Take what you’ve learned and begin again

Rather than qualitative vs. quantitative, the most clever analysts use the two together.

Quantitatively identifying problems in aggregate and qualitatively understanding the underlying behaviors behind those problems has become a critical workflow (that, as you guessed, works in both directions!) that drives product improvements:

• For Wistia, user confusion about how to use the product was reduced through allocating additional resources to customer support.

• Metromile realized that they needed to add educational content to their homepage to reduce user confusion and encourage more quote requests.

• Kayako learned that they needed to overhaul their user onboarding, and designed a completely new onboarding flow to guide users through the part of the product they were most interested in.

• SpanishDict increased engagement by moving the ad banner that was blocking the search bar to optimize for mobile users.

Ultimately, this results in a product feedback loop.

The quantitative-qualitative flywheel ensures your brain will never have to scramble to fill in gaps. The best qualitative and quantitative analytics tools work together to reveal the full story of user behavior on your site.

Put the flywheel to work and reap the rewards of an improved Digital Experience Intelligence platform.