Embedding Our New React UI in Go

We’re not far off the 1.17 release of Flipt. It comes with a fresh coat of paint and some foundational changes to the Flipt user interface. We wanted to take a moment and share part of the engineering journey taken to get to this release.

TL;DR

• We’re moving from Vue to React 🎉
• We experimented with Next.js 🧪
• We struggled with client-side routing 🗺️
• We settled on plain-old React + hash-based routing ⚛️

As we neared the end of 2022, we were settling new authentication features we had been building. The ability to support login in the UI was a primary motivator for the backend changes we were making, so naturally, we were closing in on building login functionality in the frontend. Since we had to tackle some reasonably sized frontend changes anyway, we took a moment to reconsider the state of our frontend architecture.

When Mark first laid the foundations of the current UI back in February 2019, Vue.js presented a quick and easy way to do so. Particularly, in such a way that could be easily embedded into Flipt. Flipt is a single binary built primarily in Go, and it serves both the HTTP and gRPC protocols.

The UI is served over HTTP directly from the binary using Go’s fantastic http.FileServer implementation from the standard library. Sometime around February 2021 was the Go 1.16 release, which came with native embedding of files into Go binaries. Flipt quickly took advantage of this feature for bundling the static UI assets directly into the binary. Go has a handy wrapping function that adapts the embed.FS into the required http.FileSystem interface to integrate it seamlessly into an http.Server. You can find the current (v1.16.0) embed directive in Flipt here.

Fast-forward back to today, and we're reviewing the future of the UI. It appears that React has dominated the world of frontend frameworks since Flipt inception. As such, we have speculated that React could have a broader appeal for Flipt contributors, so we have decided to take the plunge now before we get any more invested in frontend features.

The frontend is not our natural habitat at Flipt. We’re learning by doing. When we decided to take the first steps to transition from Vue to React, we looked around for what is the current hot stuff on the market. This is where Next.js caught our attention. We had some initial familiarity with it, as this blog and our landing page is built using it. Next has plenty of batteries included tooling and how-to material to sink our teeth into, so we got to work spiking a prototype.

It didn’t take long for us to get a prototype using Next.js together. Where things started to get tricky was getting Next.js to fit in our Go binary, where the existing UI currently resided.

Challenges

Challenge 1: Bundling our Next.js application into a Go binary

Our first goal was to try to replicate what we had already. Easy right? Last time we just:

1. Ran a build process in the UI project to produce a directory of assets.
2. Embedded those assets using Go’s embed tooling.

This first attempt didn’t take too long to replicate. A little digging around Next’s documentation led us to this page: https://nextjs.org/docs/advanced-features/static-html-export. After building the site with npm run build, a subsequent next export appeared equivalent to how we produced and consumed our Vue.js application. A little more digging to get the export CLI flags right, and it was as simple as next export -o $FLIPT/ui/dist.

Now we have all our assets (HTML, JS, CSS etc) in the correct directory. Without changing anything else, we booted Flipt and took a look. At first glance, everything seemed like it might have just worked.

Challenge 2: Missing Assets

However, things were not quite what they seemed. Interactions weren’t working as expected. A brief look under the hood (via developer tools), and sure enough, there were a bunch of errors.

The errors were a little misleading at first. What has this got to do with X-Content-Type-Options: nosniff? Why is text/plain being returned as the content-type for JS assets by our file server? Popping open the network tab was a little more enlightening.

These failing assets were returning 404 (Not Found) status codes, but not everything was returning a 404. Looking at the responses a little closer shined a light on our weird Content-Type and “blocked” error messages. Our Go static file server was presenting a generic 404 response page (Content-Type: text/plain) for our missing assets. These assets were being requested via script tags, and their security mechanisms were kicking in when they got back a content-type they weren’t expecting.

Long-story short, our Go file server was not serving some subset of our assets. A quick check of the asset folder we created earlier, and the assets were actually where they should be.

➜  tree ui/dist
.
├── 404.html
├── _next
│   ├── E5aRj9aVYvtSy47LN7sYl
│   └── static
│           ├── E5aRj9aVYvtSy47LN7sYl
│           │   ├── _buildManifest.js
│           │   └── _ssgManifest.js
// trimmed output for brevity

So, what was it that these particular assets had in common, separate from the assets which were being served successfully?

Could it be that they start with an underscore? A bit of digging into Go’s embed standard library turned up just what we were looking for.

If a pattern names a directory, all files in the subtree rooted at that directory are embedded (recursively), except that files with names beginning with ‘.’ or ‘_’ are excluded.
…

If a pattern begins with the prefix ‘all:’, then the rule for walking directories is changed to include those files beginning with ‘.’ or ‘_’. For example, ‘all:image’ embeds both ‘image/.tempfile’ and ‘image/dir/.tempfile’.

It appears that all we needed was to update our call to the embed directive to the following:

-// go:embed dist/*
+// go:embed all:dist/*
var UI embed.FS

This did the trick. At-least, the errors in the console were gone. All assets were returning status code 200.

Challenge 3: Right-Click, Open New Tab

It didn’t take long to stumble onto the next set of issues and subsequent can of worms. When attempting to open a link in a new tab, or visiting it in a new window, we were encountering errors.

Some actions returned 404s, and some listed links to other pages with unexpected content.

What is [pid].html? Why are we seeing 404s attempting to create a new flag? I thought this was simply another single-page app? Are we actually navigating to non-existent pages?

Comparing old and new side-by-side, we spotted something we had not considered during the transition. The routing appeared to be different.

We spotted the # before the path in our existing Vue.js application. The hash in the URL path (known as a URI fragment) is never sent as part of a server-side request. It is purely there to serve the frontend. When added to a URL, most browsers will attempt to locate an equivalent HTML ID attribute in the page and scroll to that location in the view. However, it has since become popular to leverage the hash value to drive single-page application paths.

In reality, only the root index.html (the file server for / path) file is being served.

In hindsight, that makes sense. It is a single-page application. However, in our new application, we have real paths in the URL. These appear to work as expected when you stay within the same window, but the moment you navigate from a new context, we see attempts to serve these paths from the Go file server.

This is where the differences in the transition really started to become clear. We put on our thinking caps to explore a path forward.

Unsurprisingly, as the dust settled, some of these problems and solutions started to present themselves in other documentation and blog posts. That was after some failed attempts to Google the problem and come up with our own hacks solutions. Fortunately, we were on the right path.

We explored and prototyped three different paths forward.

1. Update our Go file server to only serve index.html for all (HTML asset) paths.
2. Teach our Go file server how to handle Next.js File-System Routing.
3. Change back to hash-based routing in our frontend application.

Serving index.html for all HTML requests

One quick trick (which we later found outlined in Vue’s documentation on a similar subject) is to serve the index.html page generated by Next, whenever any request for HTML is made.

path requested	currently rendered	desired page
/flags	/flags.html	index.html
/segments/one	/segments/one.html	index.html
/console	/console.html	index.html

You get the idea.

Though this isn’t the full picture as one small additional change is required. By default, it appears that the Next.js router doesn’t attempt to reconcile the actual browser path (window.location.path) with the routers' initialization value for the path. When we render index.html for say the /segments page, the router initializes with the path /. I suspect our particular use case is something Next.js isn’t trying to cater for.

In order to get around this problem, we employed a quick hack. On page load, we forcibly update the router's path if it was not as expected via a useEffect. This causes a little visual hop from the root page to the desired destination... but it works. If you know of a better way of doing it, come nerd-snipe us in our Discord.

Implementing Next.js File-System Routing in Go

This approach was a little heavier on the Go side. The crux of the solution is to support Next.js ability to serve dynamic pages via an on-disk naming convention using square brackets in the filename (This is our friend [pid].html).

Firstly, there is one-quick trick we did with next export functionality, that saved us some further complexity in our implementation. Set the trailingSlash property in the Next configuration to true. https://nextjs.org/docs/api-reference/next.config.js/trailing-slash

This changes the exported output structure of the static HTML slightly.

• segments.html becomes segments/index.html
• [pid].html becomes [pid]/index.html.

This happens to be more congruent with how the Go file server handles particular request paths. When a path happens to refer to a directory, like /segments is in Flipt, Go will render an HTML snippet containing a list of links to the directories contents. This is what led to the screenshot earlier. Containing the following files:

• [pid].html
• new.html

When that directory happens to contain an index.html, it serves that instead.

This gets us a lot of the way there, however, the final challenge is dynamic routes in Next. For example, when requesting the path /flags/my_flag_key, the standard Go file server will attempt to open the following two files:

• /flags/my_flag_key
• /flags/my_flag_key/index.html

Neither of these files exists, as the my_flag_key refers to a flag key stored within the Flipt backend system. With next.js, what we really want to serve is /flags/[pid]/index.html. This page knows how to handle extracting the key from the path and requesting it from the backend.

We need a way to map all dynamically routed paths to their respective dynamic files “on-disk”. I say “on-disk” in inverted quotes because remember, we embed these files into our Go binary earlier. The assets are not really on-disk. They are exposed via a filesystem abstraction (an interface) in Go.

Go’s embed.FS is a structure that implements the fs.FS interface. This interface requires a single Open(name string) (fs.File, error) method.

Given that we know and can explore the contents of this directory on initialization of Flipt; we do so to derive the expected dynamic paths. Then we implemented a decorator of the fs.FS interface, which decides whether an incoming file path needs to be re-directed and open one of the dynamic paths.

type nextFS struct {
  // we will slot our embed.FS instance in here
  fs.FS

  // prefix tree of potential paths
  tree *node
}

func newNextFS(embedded fs.FS) fs.FS {
  var root *node
  // walk provided filesystem "embedded" to derive
  // potential dynamic paths and build a tree.

  return &nextFS{
    FS:   embedded,
    tree: root,
  }
}

func (*nextFS) Open(name string) (fs.FS, error) {
  var adjustedPath string

  // walk tree in order to replace any dynamic path
  // parametes with the name of the dynamic paths on disk
  // building and adjusted path to the desired file.

  return n.FS.Open(adjustedPath)
}

Our first pass implementation of this can be found here. The TL;DR is we built a crude prefix tree based on the paths in the embedded filesystem, with the dynamic files annotated within the tree.

When a requested file matches a dynamic path in the prefix tree, we re-write the path name before handing it on to the embedded fs.FS implementation.

Now /flags/my_flag serves the contents of /flags/[pid].

Embracing Hash Routing

Finally, hash-based routing. This is what our current UI does with Vue.js. Re-instating this routing capability as the primary approach to client-side routing brings us back in line with how Flipt works today. It fully encapsulates Flipt as a single-page application, meaning, we can easily embed it into a simple, vanilla file server implementation in Go. Furthermore, this approach is backward compatible with existing Flipt URLs (without additional changes).

As this potential strategy dawned on us, we also decided to review our decision to use Next.js. We still feel that adopting React is a worthwhile bet for the future of Flipt, however, we don’t have to use Next.js to do so. In fact, we suspect vanilla React will be more broadly understood than Next.js right now. Next.js brings plenty of extra opinions, which in hindsight are not necessarily congruent with where Flipt is going.

Finally, React Router has an out-of-the-box implementation of hash routing right there for the taking.

The Decision

After exploring all of the above, we settled on approach (3), vanilla React and hash-based routing.

Sorry, perhaps next time Next.js. You’re very cool and shiny. But React is all we require right now.

The product of this choice is no change to the Go side of the equation. The frontend application involves a simple npm-driven build process, which outputs a “dist” folder containing all the necessary assets. This folder contains a single HTML file (index.html), along with several Javascript and CSS files. We embed the folder exactly as we had before, using the Go standard library to both embed and serve it.

Final Words

We hope that moving from Vue to React will attract more contributions from a wider pool of developers. Flipt is an open-source solution, and we want to build an active community around it.

The new UI will be part of the v1.17.0 release of Flipt. We hope you will take a moment to give it a test drive. Hot on this release's heels will be login functionality in the UI.

We intend to publish another blog post around v1.18.0 talking more about the journey to this new UI, the new authentication related features and where we’re going next.