engineering blog-dk-purple-8-1200x628
Engineering · 8 min read

Build your own automated sync bot via GitHub Actions

Fullstory is a Digital Experience Intelligence platform that recreates user interactions on websites and native mobile apps to provide our customers with insights on where they can improve user experience. We strive to create an easy implementation path for developers so that they can get Fullstory running on their sites quickly. To that end, initializing Fullstory is as easy as copying/pasting several lines of JavaScript code (called “the snippet”) into the website. 

This works very well for traditional HTML sites, and is also pretty simple via a Content Management System (CMS), Tag Manager, or eCommerce platform. However, if the website is a Single Page Application (SPA) built using frameworks like React, Angular, Vue, etc., this path is less than ideal. We realized that we needed to provide an idiomatic way to add the recording snippet and our client JavaScript APIs to SPAs. So, we built an open source NPM package: the Fullstory Browser SDK.

Maintainability of the snippet in the browser SDK

Because we provide installation instructions both in our web app and also programmatically in the SDK, the core snippet code is now hosted in two places: in our closed source repository and on an open source repository that contains the code distributed via NPM. So, how do we keep this critical piece of code in sync across two repositories?

If we were to do this manually, we would have to remember to update the open source repo whenever we release a new version of the snippet. This would involve two separate teams (the team who maintains the snippet source code, and those who maintain the open source repo) to coordinate their work. And if you frown at that, so did we. We wanted an automated way to keep the open source code in sync with the snippet code inside the closed-source repo.

An automated solution built on GitHub Actions

We needed a good way for our internal microservices as well as the open source consumers to share a single source of truth without being tightly coupled. We initially considered a “push” model where our build system would push any snippet updates to the open source repo and create a PR. The drawback is that it would require our build system to maintain secrets and understand the structure of our open source repo, and to maintain a repository that it does not own. These added complexities and dependencies in the build system are not ideal.

Eventually, we came up with a solution that relies on two technologies:

  • A new public API that we host which returns the snippet code

  • GitHub Actions

In adding GitHub Actions to our open source repo, it was trivial to use the built-in cron function to pull the latest snippet code that we expose via the public API endpoint. All we needed was to create a new microservice to serve the snippet for a variety of different clients to consume. Moreover, GitHub Actions has built-in git operations and GitHub management APIs, which make it easy to automatically update the snippet file, check out a new branch, open a PR, and assign it to the correct reviewers.

A tour inside the Snippet-Sync-Job

Step 1. Building the snippet service and exposing a public API endpoint

In order to be able to pull the latest snippet from our closed-source repos, we needed to expose a public API serving the snippet code. It serves up either a “core” or “ES Module” version of the snippet based on the URL parameters passed in. And voila! We have a way to pull the latest snippet. It’s worth noting that the service is used by various other services we host internally via gRPC as well.

github-actions-image-1

Step 2. Adding Github Actions to our open source repo

To enable GitHub actions, first create a main.yaml file inside .github/workflows folder. And with just a few lines, we can define a cron job and pull for updates every 24 hours.

GitHub actions make it trivial to authenticate in a workflow: it provides a GITHUB_TOKEN secret, and several other default environment variables. Along with the snippet API endpoint, we now have all we need to proceed.

We will use all the imported environment variables and the parsed repoInfo later on.

Step 3. Check for snippet updates and existing open PRs

In the Sync snippet step, we’ve already checked out the latest main branch. We first use axios to pull the latest snippet text via REST API, then compare the hash of the latest snippet text with the one we have on the file system of the checked out repo. We continue to the next steps only when we find the mismatch in the hash values, meaning a snippet update has been detected.

If we determine that an update is needed, we initialize an octokit client, which is provided by the @actions/github package. The client is authenticated using the GITHUB_TOKEN env var declared in the main.yaml file. 

We then get the list of current open PRs and check to see if the same PR has already been created. Since we run the sync job every 24 hours, it’s possible that an existing PR has been opened but not yet merged, in which case we do not want to open another PR. We achieve this by simply looking for a PR created by the Github Actions bot, and that the title is a constant:  “The Fullstory snippet has been updated.”

Step 4. Use Github octokit to obtain the tree object

The next step is to update the snippet.js file and create a PR.

In order to programmatically update the file and open a PR, we needed to get deeper into what git calls the Plumbing commands. If you are not familiar with the inner workings of git, we recommend following the above hyperlink to read about it before continuing 

The Tree Object is the data structure that holds the relationships between your files (blobs), similar to a simplified UNIX file system. We need to first create a new Tree Object with the new contents of the snippet code. And then use the newly created tree to checkout a branch and open a new PR.

To do so we need to first get the current commit. At this point we’ve checked out main and we’ve already obtained the current commit sha from process.env in Step 1. With the commit sha we can now get the current commit via octokit.git.getCommit, which contains the hash of the tree object: tree sha. With the tree sha we can then get the tree object via octokit.git.getTree with a recursive parameter.

Once we got the tree object, we then found the “tree node”(srcTree) with our known SNIPPET_PATH. It’s a tree node that represents the snippet.js file.

Step 5. Create a new Tree with modified content

Let’s summarize what we have so far:

  • We have the new snippet from the public API hosted by Fullstory, in string format

  • We have the source “tree node” that holds the content of the snippet from the current commit on the main branch

The next thing we need is to create a new tree object with the new content (new snippet text). To do so we create a new tree with  octokit.git.createTree and specify the updated object: our new snippet text. Remember that we’ve retrieved the original tree object recursively, meaning the tree object contains references to all the files with their nested paths. The new tree will contain all the information in the original tree, but update only what we need: the snippet.js file

github-action-sync-bot-image-2

Step 6. Commit the change to a new branch

Now that we have a new tree object with updated snippet text, it’s simply a matter of committing the change and opening a PR. 

With the current commit as parent, we create a new commit using octokit.git.createCommit and pass in the created tree’s tree sha, then create a new reference (branch) with the name: snippetbot/updated-snippet-${Date.now()} using octokit.git.createRef, providing it the commit sha we just created:

Step 7. Open a PR and assign it to the maintainers

The final step is to create a PR via octokit.pulls.create and assign it to the correct maintainers of the repo via octokit.issues.addAssignees.

In order to get the correct assignees, we maintain a MAINTAINERS.json file that contains all the maintainer’s GitHub handles for this repo, so the correct team members will be notified to review and merge the new PR.

Results and closing thoughts

With GitHub actions we were able to automate a process that would've been cross-team, manual, and error prone. Our automation enables each team to operate independently, helps improve our productivity, and simplifies the process of maintaining the open source project for our Browser SDK.

It also ensured that our consumers of the NPM package would have the latest version of our snippet to take advantage of any new features we release.

Moreover, we now have a pattern for syncing our closed source code to the open source repos without depending on any human intervention. 

With this architecture, we’ve detected and merged several updates since it came online in December 2019, ensuring updated code for our Browser SDK users and reducing maintenance overhead.

Check it out in action (and learn more about the Fullstory Browser SDK) here.

Want a perfect website or app? Fullstory can help. Request a demo today.

author

Sabrina Li

Software Engineer

Sabrina is a software engineer on the Ecosystems team at Fullstory. She is based in Atlanta, Georgia.