Fullstory is a Digital Experience Intelligence platform that recreates user interactions on websites and native mobile apps to provide our customers with insights on where they can improve user experience. We strive to create an easy implementation path for developers so that they can get Fullstory running on their sites quickly. To that end, initializing Fullstory is as easy as copying/pasting several lines of JavaScript code (called “the snippet”) into the website.
This works very well for traditional HTML sites, and is also pretty simple via a Content Management System (CMS), Tag Manager, or eCommerce platform. However, if the website is a Single Page Application (SPA) built using frameworks like React, Angular, Vue, etc., this path is less than ideal. We realized that we needed to provide an idiomatic way to add the recording snippet and our client JavaScript APIs to SPAs. So, we built an open source NPM package: the Fullstory Browser SDK.
Maintainability of the snippet in the browser SDK
Because we provide installation instructions both in our web app and also programmatically in the SDK, the core snippet code is now hosted in two places: in our closed source repository and on an open source repository that contains the code distributed via NPM. So, how do we keep this critical piece of code in sync across two repositories?
If we were to do this manually, we would have to remember to update the open source repo whenever we release a new version of the snippet. This would involve two separate teams (the team who maintains the snippet source code, and those who maintain the open source repo) to coordinate their work. And if you frown at that, so did we. We wanted an automated way to keep the open source code in sync with the snippet code inside the closed-source repo.
An automated solution built on GitHub Actions
We needed a good way for our internal microservices as well as the open source consumers to share a single source of truth without being tightly coupled. We initially considered a “push” model where our build system would push any snippet updates to the open source repo and create a PR. The drawback is that it would require our build system to maintain secrets and understand the structure of our open source repo, and to maintain a repository that it does not own. These added complexities and dependencies in the build system are not ideal.
Eventually, we came up with a solution that relies on two technologies:
A new public API that we host which returns the snippet code
GitHub Actions
In adding GitHub Actions to our open source repo, it was trivial to use the built-in cron function to pull the latest snippet code that we expose via the public API endpoint. All we needed was to create a new microservice to serve the snippet for a variety of different clients to consume. Moreover, GitHub Actions has built-in git operations and GitHub management APIs, which make it easy to automatically update the snippet file, check out a new branch, open a PR, and assign it to the correct reviewers.
A tour inside the Snippet-Sync-Job
Step 1. Building the snippet service and exposing a public API endpoint
In order to be able to pull the latest snippet from our closed-source repos, we needed to expose a public API serving the snippet code. It serves up either a “core” or “ES Module” version of the snippet based on the URL parameters passed in. And voila! We have a way to pull the latest snippet. It’s worth noting that the service is used by various other services we host internally via gRPC as well.
Step 2. Adding Github Actions to our open source repo
To enable GitHub actions, first create a main.yaml
file inside .github/workflows
folder. And with just a few lines, we can define a cron job and pull for updates every 24 hours.
GitHub actions make it trivial to authenticate in a workflow: it provides a GITHUB_TOKEN secret, and several other default environment variables. Along with the snippet API endpoint, we now have all we need to proceed.
We will use all the imported environment variables and the parsed repoInfo
later on.
Step 3. Check for snippet updates and existing open PRs
In the Sync snippet
step, we’ve already checked out the latest main
branch. We first use axios to pull the latest snippet text via REST API, then compare the hash of the latest snippet text with the one we have on the file system of the checked out repo. We continue to the next steps only when we find the mismatch in the hash values, meaning a snippet update has been detected.
If we determine that an update is needed, we initialize an octokit client, which is provided by the @actions/github package. The client is authenticated using the GITHUB_TOKEN
env var declared in the main.yaml
file.
We then get the list of current open PRs and check to see if the same PR has already been created. Since we run the sync job every 24 hours, it’s possible that an existing PR has been opened but not yet merged, in which case we do not want to open another PR. We achieve this by simply looking for a PR created by the Github Actions bot, and that the title is a constant: “The Fullstory snippet has been updated.”
Step 4. Use Github octokit
to obtain the tree object
The next step is to update the snippet.js
file and create a PR.
In order to programmatically update the file and open a PR, we needed to get deeper into what git calls the Plumbing commands. If you are not familiar with the inner workings of git, we recommend following the above hyperlink to read about it before continuing
The Tree Object is the data structure that holds the relationships between your files (blobs), similar to a simplified UNIX file system. We need to first create a new Tree Object
with the new contents of the snippet code. And then use the newly created tree to checkout a branch and open a new PR.
To do so we need to first get the current commit. At this point we’ve checked out main
and we’ve already obtained the current commit sha from process.env
in Step 1. With the commit sha we can now get the current commit via octokit.git.getCommit, which contains the hash of the tree object
: tree sha
. With the tree sha
we can then get the tree object via octokit.git.getTree with a recursive parameter.
Once we got the tree object, we then found the “tree node”(srcTree
) with our known SNIPPET_PATH
. It’s a tree node that represents the snippet.js
file.
Step 5. Create a new Tree with modified content
Let’s summarize what we have so far:
We have the new snippet from the public API hosted by Fullstory, in string format
We have the source “tree node” that holds the content of the snippet from the current commit on the
main
branch
The next thing we need is to create a new tree object
with the new content (new snippet text). To do so we create a new tree with octokit.git.createTree and specify the updated object: our new snippet text. Remember that we’ve retrieved the original tree object recursively, meaning the tree object contains references to all the files with their nested paths. The new tree will contain all the information in the original tree, but update only what we need: the snippet.js
file
Step 6. Commit the change to a new branch
Now that we have a new tree object
with updated snippet text, it’s simply a matter of committing the change and opening a PR.
With the current commit as parent, we create a new commit using octokit.git.createCommit and pass in the created tree’s tree sha
, then create a new reference (branch) with the name: snippetbot/updated-snippet-${Date.now()}
using octokit.git.createRef, providing it the commit sha we just created:
Step 7. Open a PR and assign it to the maintainers
The final step is to create a PR via octokit.pulls.create and assign it to the correct maintainers of the repo via octokit.issues.addAssignees.
In order to get the correct assignees, we maintain a MAINTAINERS.json
file that contains all the maintainer’s GitHub handles for this repo, so the correct team members will be notified to review and merge the new PR.
Results and closing thoughts
With GitHub actions we were able to automate a process that would've been cross-team, manual, and error prone. Our automation enables each team to operate independently, helps improve our productivity, and simplifies the process of maintaining the open source project for our Browser SDK.
It also ensured that our consumers of the NPM package would have the latest version of our snippet to take advantage of any new features we release.
Moreover, we now have a pattern for syncing our closed source code to the open source repos without depending on any human intervention.
With this architecture, we’ve detected and merged several updates since it came online in December 2019, ensuring updated code for our Browser SDK users and reducing maintenance overhead.
Check it out in action (and learn more about the Fullstory Browser SDK) here.