Introducing the new privacy-aware digital experience product for mobile apps from Fullstory
Hi there. I’m Matt, and for the past five years, I’ve been working with some of the best engineers in the world to develop Fullstory for Mobile Apps: our first product for understanding how users interact with mobile app experiences.
I have been active in the mobile space since the days of the Palm Pilot. My first foray into the world of the “smartphone” was with the Treo 650—believe it or not, back then “smart phones” didn’t always come with wifi capabilities and data plans were incredibly expensive.
Yes, this is how we used to get phones on wifi.
But times have changed. Today, as is the case for so many of us, my phone is my “outboard brain.” It holds the entirety of my planning, my notes, and my half-formed thoughts. With everything from email drafts, to medical notes for myself and my family, passwords, and sensitive financial information, my phone is my castle.
Working on mobile app experiences at Fullstory has given me a unique opportunity to make the world Structured Wrongness with the Suck Less Cycle for myself and the people around me in multiple dimensions.
As a mobile engineer, I’m hyper aware of the shortcomings of existing technologies. Historically, mobile app teams have had to cobble together quantitative analytics, crash reports, and support tickets to understand what issues are impacting their end users. Now, with Fullstory for Mobile Apps, these teams can quickly discover, understand, and fix user-impacting digital experience issues.
Most importantly, as a mobile user, a father to a young kid, and a citizen—I’m obsessed with the privacy environment we’re building for the future. And at Fullstory, we are working to provide critical visibility into the mobile experience while holding end user privacy preeminent.
Mobile Experience Diagnostics Have Been Historically Underwhelming
When the Fullstory founders approached me to prototype a digital experience product for mobile apps, I was excited. The need for usable mobile experience analytics was (and is) huge.
The reality is that Leveling up your mobile experience design. From my banking app failing to sign me in, to my insurance company’s app cutting off that one critical bit of information on a screen, to the airline experience that was less-than-perfect, to my son’s transit app—which frankly just doesn’t work at all… all of these bad mobile experiences are like death by a thousand cuts.
Clearly, companies across industries are missing the mark when it comes to meeting consumer expectations. Source: PwC (2018)
And in this moment, the mobile experience is more critical than ever. So much is unknown. We are stressed out, anxious, afraid, trying to find footing on shifting sand. People need the information they need, now. They need the products they need, fast. And they are looking to their mobile devices to deliver.
In bringing Fullstory to mobile apps, I knew we had an opportunity to impact the quality of life for a lot of people: End users and mobile product owners and engineers, alike.
The “mobile experience analytics” space has been developing for some time, but has never been great. Today mobile apps are shipping with multiple analytics libraries to try to understand the cowpaths users are wandering.
Some existing products in the space attempt to give rich visual analytics by recording end users' screens. Besides capturing far more information than they need, these products take up a ton of bandwidth—which limits use cases. They might be useful, for instance, if you wanted to follow up on a limited set of beta users that were guaranteed to be on wifi, but you couldn’t go wider.
Privacy controls have also been rudimentary and far too easy to get wrong. If you do get them wrong—for example, collecting sensitive data like credit card numbers that you never intended to capture—there hasn’t been an easy way to fix the mistake, other than the “nuclear option” of deleting everything.
We wanted Fullstory for Mobile Apps to raise the bar in terms of the quality of the product as well as our privacy-first approach.
We're Determined to Get Privacy Right
When we kicked off this effort, we knew we wanted to do things differently. We set out to build a product for Android and iOS that we deemed worthy—that had the fidelity and quality that we could be proud of.
By the beginning of 2019, we had developed a powerful analytics library which allowed users to define searches, create funnels, and understand issues through session replay. We were able to leverage the same selector syntax as our web product and session playback fidelity surpassed our expectations, all while using an order of magnitude less bandwidth than any other solution on the market.
Yet even while we had accomplished the goals we established when we set out to support native mobile apps, we didn’t release the functionality.
Throughout the process, we had been testing the product internally using real-life applications. These apps housed sensitive data ranging from mildly sensitive data—such as musical tastes—to deeply private data like financial information. We realized that replay fidelity and low-bandwidth were just two facets of the problem we wanted to solve.
Looking at the big picture, we wanted our friends and family—and every mobile app user—to know that companies that use Fullstory are treating their digital information with the utmost care and respect.
While the first version of the product featured the same exclusion rules you’d see in other products today, our consensus was that we absolutely had to build more privacy tooling before we could ship it.
To that end, the Fullstory team focused on designing a privacy solution for our mobile product that would make privacy first-and-foremost and give our customers a set of building blocks they could use to get privacy right.
The result is a mobile analytics platform that we consider to be both novel and best-in-class. We’re inviting everyone in the space to take a look at our approach and think about how we can all set the bar with end users in mind.
Lifting the Hood on Mobile Capture
Past approaches in the digital experience space have used one of two different techniques:
Capturing mp4 video, or
Capturing a sequence of JPEG/PNG images
As a mobile engineer or product manager, your choices were either high-fidelity/high-bandwidth playback at the expense of the end user’s data plan, or a low-bandwidth, blurry representation that often couldn’t accurately illustrate the paths end users were taking. But why were those the only two options?
Image and video codecs are great for compressing photos and videos, but they are not designed to store the highly-detailed, pixel-perfect nature of user interfaces. By repurposing those codecs, the industry was able to capture the contents of a screen. But this came at the expense of wasted bytes and CPU cycles pretending that they were capturing reality.
The question we asked ourselves was: What if we could skip the codec and just re-draw the same things that were visible on-screen?
Starting With the Tree
Like HTML documents in the browser, mobile applications on Android and iOS are composed of a hierarchy of dozens to hundreds of views. Those come with basic properties that describe where the view appears on-screen, what type of view it is, and additional metadata: identifiers, accessibility information, and more.
Our process begins by indexing the complete hierarchy and metadata at regular intervals while the application runs. But this information only gets us to a basic application wireframe, which is useful for high-level, structural analysis but not much more.
Completing the Picture
Every on-screen view in a mobile app is composed of a number of lower-level drawing commands: shapes, paths, bitmaps, text, and a variety of others. Here’s an example of how one kind of view might break down:
If you were to take a screenshot of that individual view, you might end up with 10 to 20 kB of image data to upload. In experiments we’ve run locally, this can translate to megabytes per minute. That’s far too much data just for analytics!
Instead, we are able to encode the fundamental drawing operations as tightly as possible—using as little as tens of kilobytes per minute. All this while still maintaining great fidelity.
Fullstory for Mobile Apps lives in the Ideal Zone.
This encoding method gives us access to the fundamental drawing operations on mobile devices, which allows us to address privacy on a much finer scale than before—with surgical precision.
"Solving mobile issues can be extremely difficult. The amount of detail needed to solve an issue can take a lot back and forth with the customer, and even then it can be a challenge to reproduce. Since implementing Fullstory on our Native Mobile apps, we have seen this resolution time go from months to days to fix some of our biggest mobile issues." — Joshua Lowe, Product Manager at CINC
The Technical Side of Privacy
Fullstory for Mobile Apps is based on a view hierarchy, enhanced with drawing operations. Because we have access to the underlying drawing commands, we are able to replace all textual information with an approximation of the underlying text bounds.
The result is a skeleton of the app structure—the hierarchy of views and visual clues to layout—wherein potential sensitive text information is redacted.
We do something similar for image content. Any bitmap drawing operation we see can be replaced by a placeholder single-color image like so:
Fullstory is able to extract enough of the app experience to illuminate how users flow through the app, to surface frustration signals, and to glean insights into users' most valuable paths—without capturing information we don’t need.
The Balanced Path
There’s a fine line between giving engineers and product managers the data they need to identify, diagnose, and address subpar mobile app experiences without sacrificing end user privacy in the process. We believe that giving our customers the ability to walk that knife’s edge with confidence is non-negotiable.
We are constantly thinking about “comfortable analytics”—that is, analytics that users feel is appropriate for application developers to capture to make their mobile experiences better. More specifically, analytics that consider the real human in the mix and how they feel about their data.
Our goal is to give our customers a spectrum of tools to help them navigate this narrow space of “comfortable analytics” and collect only what’s necessary to continuously improve.
The privacy tool spectrum.
In terms of data collection, we operate with a Private by Default approach. This means that Fullstory masks from collection—by default—all text and images that are displayed on the user’s screen.
Practically speaking, we are creating a wireframe of the app. This allows our customers to see their end user’s interactions, but no on-screen data is collected or sent to Fullstory unless it has been explicitly unmasked.
Masking, unmasking, and exclusion rules give our customers unparalleled control over which data is captured—either in code or via our web-based configuration tooling. On top of this, we’re shipping a new “privacy preview” to quickly allow a Fullstory user to visualize the effect of making changes to their privacy rules.
Fullstory users can visualize the effect of different privacy rules.
In the unlikely event that sensitive data is unintentionally collected, data-loss prevention (DLP) algorithms will stop some sensitive data, such as credit card or personal identification numbers, from leaving the end user’s device.
If information does leave the device, our server-side redaction process gives us the ability to surgically redact only sensitive data from previously captured sessions and audit which Fullstory users have seen it. We’ve also built a comprehensive set of final-resort tools to delete all of the data from a set of sessions potentially containing accidentally-captured sensitive data.
Our partners in the ecosystem are also helping us build patterns and best-practices in the privacy space so that we can ensure that everyone using Fullstory is doing so in a responsible way. Our Fullstory now SOC 2 Type II certified certification demonstrates our commitment to following processes to keep data private and secure.
Fullstory for Mobile Apps is the first digital experience product that is both privacy-conscious and bandwidth-preserving enough to enable for 100% of end users.
We are uniquely positioned to impact the baseline level of privacy for an entire product segment (and heck yeah we’re going to push to make sure that everyone else gets this right).
What About the Web?
Note that as we worked on building this out for mobile experiences, our team realized the importance of shipping the same functionality for browser-based sessions.
Private by Default for the web has been in the works for the last year. And I’m happy to say that we are now shipping the same privacy tooling for customers who want to improve their website experiences. It's available via request today, and we will be helping all of our customers enable it over the coming months.
We’ll publish more in-depth technical details about the web solution in the coming weeks on this blog—so stay tuned!
Keeping Our Outboard Brains Safe
I’m writing this post because I’m confident that we have built a solution that will push the industry forward, a solution that clearly demonstrates the Empathy, clarity, bionics: the Fullstory watchwords under which all of us at Fullstory operate. But also because I want to invite dialogue with all of you.
Given the potential reach of the code we are writing, I am continuously assessing whether I would be comfortable with this product running on the devices of the people in my life—and I encourage those on my team to think in the same way.
I am constantly checking in, asking if we are striking the right balance between helping our customers provide better mobile experiences and protecting the sanctity of our “outboard brains.” But this is an industry-wide, society-wide effort. Our goal is to keep challenging and pushing everyone in the space to build analytics that consider the real human.
Curious to Learn More?
Curious about how Fullstory for Mobile Apps works? See potential to use Fullstory for Mobile Apps on your own app? Intrigued by the conversation around privacy?
Give us a shout—we'll walk you through the product and demonstrate how you can enable privacy-aware analytics on your web or mobile application.
