How even the most tantalizingly beautiful data visualizations can go completely wrong
Wouldn't it be incredible if you had a way to quickly see how your aggregate user base engaged with your website or app? Well, heat mapping tools claim they can do just that.
But can heatmaps deliver the goods? And at what cost to the reliability, accuracy, and actionability of the information? What are heat maps, really, anyway?
What are heat maps?
Heat maps are a data visualization that aggregates user behaviors on a website or app by applying a semi-transparent overlay using colors—red, yellow, orange, green, etc.— to the screen being analyzed. Heat maps vary color intensities to convey the intensity of user interactions, attention, or eye-movement.
Visually, heatmaps can take many forms:
While heat maps are simple enough in theory, in practice they are complex.
Consider these challenges.
How do analytics tools even know which elements and areas users are paying attention to, clicking on, or mousing over? There's no common approach. Heat mapping tools might determine areas of user interaction via eye-tracking, session replay, or perhaps just aggregated clicks on links.
While heatmaps have become a go-to visualization for showing digital user interactions, they are far from perfect. And perhaps it's not surprising that when it comes to heatmaps, the devil's in the details.
When is a heat map not a heat map? The history.
Let's back up for a second to understand how we got here. The term “heat map” or “heatmap” was trademarked by Cormac Kinney in the 90s to refer to a 2D financial market visualization (Wikipedia; also Kinney’s lapsed Trademark). Today, what most of us think of as heat maps—as evidenced by our simple definiton and sample heatmaps above—is entirely different.
According to Kenneth Field (a.k.a. Cartonerd) in his post When is a heat map not a heatmap, most of the data visualizations referred to as “heat maps” are not actually heat maps. They are technically isarithmic density maps.
What is an isarithmic density map? We'll get there, but first it's useful to take out the "density" and speak to just isarthmic maps, which the Geographic Information Systems encyclopedia defines as:
A type of thematic map that represents a continuous field using isolines (lines of equal value) or isopleths (regions of similar value). They are sometimes referred to as a heatmap, although a heatmap is only one type of isarithmic map that represents density.
A popular form of isarithmic map is a contour line map. Think about topographic maps—each connected line in a topographic map represents the same elevation:
Your run-of-the-mill topographic map uses bright line data points — elevation at specific, unique geographic locations — and interpolates (i.e. connects the dots) the missing information across the data points.
That brings us to web application isarithmic density maps—a.k.a. what we all think of as heatmaps. Web heatmaps don’t have isolines. Instead web heatmaps use isopleths. Isopleths are used to represent regions of similar value—red, orange, and yellow, green and blue color splotches.
The splotches of color on heatmaps represent aggregated user experiences on a website or application. As data visualizations go, heatmaps make for an easy-to-grok, pretty picture. You look at heatmap analyses and say, “Ah, I see!” You feel like you understand something deep and meaningful—like you're peeking behind the curtain and seeing a remarkable bit of web analytics.
But there’s a catch—or at least four catches—and if you’re hypnotized by the glowing hotness of heatmaps, you very well might miss these problems completely.
Heatmaps 101: What is a heatmap?
Find out what heatmaps are, how they started, how they’re created, and how you can use them to make data-driven decisions
4 major problems with website heatmaps
When it comes to heat maps, here are at least four major problems that you should consider because they call into question the very reliability of the data.
1. Lies, damned lies, and [X, Y] positions
In a world of responsive design and device heterogeneity, using a visualization designed for a fixed unchanging aspect ratio is a recipe for failure. Capturing engagement with an area “100px down and 350px to the right” on your site doesn’t mean the same thing across different user visits—each of whom might be browsing from different window sizes and/or screen resolutions.
Ignore these differences and you might find yourself at a confusing dead end. Responsive UI design and varying display PPIs also make converting between aspect ratios possible only when you make a lot of assumptions.
Heat mapping tools get around this problem by forcibly siloing datasets. You only see heatmaps with aggregate data for users with the exact same screen resolution configuration. Users with a viewport 1024×768 go here. Users with a viewport 1366×768 go there. 1280×800 users go in the cupboard. These arbitrary silos mean you aren’t actually seeing a meaningful aggregate across your actual non-same-screen-resolution-using users. Kinda defeats the purpose of the “easy” visualization, right?
And this is just on desktop. There are thousands of different mobile browser resolutions in the market. Most heatmap visualization solutions pretend these differences don’t matter.
Yet they matter a lot.
Average and aggregate mouse hovers and un-contextualized [X, Y] click positions cause well-intentioned product teams to make actual product changes based on shoddy information, wasting time and money, harming customer experience and negatively affecting the business.
2. Dynamic applications are the norm today
Today, a single web page is no longer a static fixed document. Modern web pages and applications are dynamic. Animations, slide out panels, expanding menus and modal dialogs are the norm. As you might guess, this poses a problem for heatmaps.
Simply put, heatmaps don’t work well with dynamic applications where the page changes. And that is an especially huge problem for JavaScript heavy single-page applications.
It isn’t just JavaScript heavy applications that are a challenge. Even server rendered websites that have data-driven content have this problem. The location of the thing you click on varies by the whims of the cosmos, popular culture, your scroll position, and what article grandma happened to post on her social media wall that day.
You end up with aggregated positional spaghetti.
And this same issue affects everything from product inventory tables on an ecommerce site to data-driven menu options in your business enterprise app.
3. Splotch analysis is botched analysis
Even when the stars align and heatmaps provide accurate and meaningful aggregations of user interactions, the visualization itself doesn’t directly tell you what you probably want to know. Because those aggregated splotches often obscure the very thing that users interacted with. You have to read the heatmap tea leaves and interpret the splotches to figure out what you think users were engaging with—what button, link, or control does that blob correspond to?
You might call this effort “splotch to element mapping:”
After seeing a heatmap and completing your back-of-the-envelope “splotch mappings,” what do you do next?
Here, heatmaps quickly turn into "analytics theater." Analytics theater is when an analysis—via a graphical visualization, chart, graph, metric, ect.—dazzles your senses with signals indicating profound insights while in reality being either completely useless or so difficult to parse into action as to become useless.
Heat maps without any related quantitative interaction data are notoriously hard to convert into actionable business intelligence ("actionable insights") because you can't measure the potential impact of a change. If you can't measure the potential impact of a change, you can't prioritize efforts to bring change to life.
Visualizations without tangible data points often become distractions that only generate more questions and spur more arguments.
4. Did we mention the configuration?
Most heatmap solutions ask you to configure complicated URL patterns and regular expressions to tell the system how to interpret interactions on a page-by-page level.
Imagine asking someone (whether non-engineer, engineer, or developer) to configure the set of possible URL regular expressions for all the important pages in a modern JavaScript application, with complex histories and dynamic URLs with identifiers in them. It’s hard to get it right in the first place, and guaranteed to go sideways the moment something changes — and your application will change.
If heat maps are broken, how do you visualize aggregated user interactions on your site?
Click maps are heat maps, evolved
Having thoroughly examined heatmaps as a possible solution to meet our customer’s requests for a visualization of user clicks on a web application or web page, we decided heatmaps needed to evolve into something better.
That's why we built Page Insights with Click Maps—to make it easy to see how your users are interacting with your website, in aggregate, while gracefully dodging the four major problems of heatmaps.
Page Insights are resolution independent
Because Fullstory records every mutation of the DOM for every page on your site where a real user visit occurs, the platform understands the structure of your site.
Fullstory translates this into aggregations that make sense across screen resolution, device type, and even the most dynamic and varied data driven interfaces by focusing on Elements. Not just position.
Page Insights take into account what’s on (or not on) the page
If you watch a replay of a user’s session with a dynamic single page application on Fullstory, you realize that many things pop in and out of existence.
Page Insights is smart enough to account for these dynamic states. It shows you what’s relevant to the state of the page in the session you're viewing. Click analytics for a non-existent popup won’t be shown when the popup isn’t there, for example.
Page Insights are visually compelling and actionable
With Click Maps, you start with a foundation of precise, element-specific data that lets you see answers immediately. And if you have questions you want to drill into, you can search to find sessions related to the numbers. If you'd like to add a color overlay for visualization, you can do that too. But, manually mapping splotches to your web app is no longer required.
Page Insights requires zero configuration
Fullstory uses machine learning to determine which pages belong to your web site or web application. Fullstory understands which pages are the same, and which pages are different. So your insights are always relevant and contextualized. No need to do complicated configuration or setup brittle URL route regular expression voodoo. It just works out of the box.
Bonus! Pivot insights based on specific segments
Page Insights gives you aggregate analyses that respect the searches and segments you build in Fullstory, too. (If you’re not familiar, here’s an explanation of Fullstory search). This helps you answer questions like “What do visitors from my ad campaign click on first?” Or, "How do returning customers interact with our site navigation?"
Page Insights make your existing Fullstory segments even more useful.
Since this blog post was originally published in 2017, Fullstory has released a number of enhancements to Page Insights including Scroll Maps, another type of heatmap designed to measure how far users scroll on any page.
Curious to learn more? Read all about Scroll Maps in this announcement or subscribe to the Fullstory newsletter to stay in-the-loop for future updates.
