In our last post, we saw how Fullstory’s Network Capture provides valuable information to find and fix defects inside your mobile app—but we also know there’s such a thing as Too Much Information. URLs can expose private information (like tokens and passwords) that should NEVER be captured or viewable in a session replay.
From the very start of development of our Network Capture feature, it was critically important to us to provide a mechanism for blocking private information from captured network requests and responses. Towards this end, the Fullstory mobile engineering team worked closely with our Fullstory Trust team to ensure the highest safeguards of your data were put in place.
As part of this collaboration, the Fullstory Trust team introduced a rich new grammar used to express URL component parts in a manner that can be independently blocked from network traffic. These rules allow a URL to be broken down into its component parts: scheme, authority, path, query, and anchor. Utilizing these rules, a privacy configuration can be established to block any or all parts of a request URL to sanitize any sensitive information before it leaves the device. So for example, you may choose to scrub out usernames, IDs, tokens, or even passwords that may be inferred from network traffic logs.
Prioritizing privacy in our implementation was our top concern…
The initial implementation of our privacy grammar was optimized for web traffic, and thus was provided for client side use in TypeScript. While this is an acceptable approach for websites, this introduced undesirable complexities for iOS and Android apps.
iOS ships with the flexible JavaScriptCore framework, but running TypeScript on Android typically requires either a sizable library or launching a WebView, either of which would cause an unacceptable impact to the customer’s app. We needed a better approach.
Fortunately, Fullstory for Mobile Apps already utilizes Rust for all cross-platform code, so it became a relatively painless effort to reimplement the URL privacy engine in Rust to provide a better native experience. Beyond simplifying the implementation, this also resulted in a much more performance solution. (More on that later.)
To allow for a high degree of expressiveness, the compiled condition and redaction rules provided by the backend are output as regular expressions. Exceptions are expressed using lookaround.
There are several crates which provide implementations of regular expressions for Rust, but we needed one that supported lookaround. Apart from one research engine, most mainstream engines that support lookaround are backtracking-based, but denial of service should not be a concern because the rules and URLs are not controlled by an untrusted third party. We went with fancy-regex, which is a relatively thin layer on top of the widely used regex crate. In addition, we needed to be able to pull apart and reassemble URLs, and the URL crate is great for that job.
…but remaining performant was crucial to our success
After implementing the rules, we discovered our Rust library had increased dramatically in size. We began by observing that URLs are strictly ASCII, and one of our engineers updated fancy-regex to support an ASCII-only mode. But it didn’t affect the binary size as much as we hoped.
After doing some research, we realized both iOS and Android support their native regexes with ICU’s regular expression package, which also provided the lookaround support we required. So, we wrote a Rust wrapper for ICU regexes, regaining Unicode support for free. Accessing the ICU regexes on iOS isn’t too difficult, but on most Android versions, one must search for the regex library and open it. The libloading crate was up for the task.
Those actions reduced our library size somewhat, but it was still bigger than the baseline. After some more research, we found the relevant RFCs for a URL provide a regular expression that is capable of parsing URLs. We then realized we could take advantage of our regex library to implement sufficient URL parsing to support our privacy rules. Note that this wouldn’t have the same robustness as a URL library designed to parse ill-formed, hand-written URLs, but those URLs wouldn’t typically work in iOS or Android networking APIs, either. We decided the tradeoff in footprint was well worth the compromise.
Once we implemented this solution, we successfully returned to our baseline library size, and our end users’ privacy and bandwidth bills were saved!
In closing
We hope you enjoyed this under-the-hood explanation of how Fullstory for Mobile Network Capture was built. At Fullstory, we believe strongly that your users deserve a more perfect digital experience, but in service of this goal we never sacrifice user privacy or performance. From the beginning, these tenants were top of mind during every step of the journey, and we are sharing the details of our implementation for transparency and clarity of our approach. We hope you enjoyed this journey and check out how Network Capture can help you improvise your user experience!
Fullstory for Mobile Apps’ Network Capture is a powerful addition to any mobile development team’s toolkit. If you want to try it, adopt the newest version of the SDK in your app.
