November 16, 2023 — 10 min read
Feature Flags and OCI
In the most recent release of Flipt (v1.31.0) we added support for OCI registries as a backend for feature flag data. The new backend follows in the footsteps of our other declarative file-based backends (local, git, and object). This blog post talks a bit about OCI, its growing popularity for non-container usecases and why and how you can leverage Flipt to distribute flags as OCI bundles.
What is OCI?
OCI stands for the Open Container Initiative. Taking words directly from their landing page:
The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.
OCI currently declares and maintains three different specifications:
Each of these specifications come together to create a standard set of operating procedures for OCI-compliant container runtimes to fetch, isolate, and execute workloads.
It is probably important to clarify first that Flipt is not concerned with running container workloads. Flipt is concerned with the distribution specification (and tangentially with the image specification). OCI-compliant registries have proven extensible and valuable beyond just containerized applications. Their distribution specification includes extensibility points, which can be leveraged to store arbitrary contents with different artifact and media types.
These extensibility points make it possible for Flipt to package and publish its feature flag configuration contents via OCI compatible registries. Some OCI-compatible registries include:
- GitHub Container Registry (Fun fact: our own Mark Phelps was a lead engineer and architect of GHCR at GitHub)
- DockerHub (the OG)
- Google Artifact Registry
- AWS Elastic Container Registry
- Azure Container Registry
To ease our integration into the OCI landscape, Flipt has built new capabilities on top of a project called ORAS. To quote what ORAS is from their site:
ORAS is the de facto tool for working with OCI Artifacts. It treats media types as a critical piece of the puzzle. Container images are never assumed to be the artifact in question. ORAS provides CLI and client libraries to distribute artifacts across OCI-compliant registries.
Big shout out to ORAS for making this possible in Flipt.
Prior-art
We’re not the first and certainly won’t be the last project to leverage OCI for non-container artifact distribution. OCI has created a standard for registry APIs, which is broadly useful for an extensible set of artifact and media types. The APIs are also designed around content-addressability, which lends itself to some very desirable caching traits and security improvements.
We got the idea to leverage OCI from past colleagues (shoutout David and Marko) and further validation from a number of modern technologies already leveraging OCI for non-container image purposes. To name a few:
Helm - A package manager for Kubernetes
Since v3.8.0 Helm has supported using OCI registries as a distribution medium for Helm charts.
https://helm.sh/docs/topics/registries/
Timoni - Another package manager for Kubernetes
From inception, Timoni has leveraged OCI registries for distributing its modules.
https://timoni.sh/bundle-distribution/
Acorn Labs - A platform for running and sharing applications
Acorns are shareable bundles of configuration for containers, services, and everything in between, which are distributed via OCI-compliant registries.
https://docs.acorn.io/advanced/publishing
Nuon - A BYOC platform
Nuon leverages OCI registries for distributing their configuration artifacts.
https://docs.nuon.co/concepts/vendor-cloud#artifacts
Improving Scale
The Git backend for Flipt provides an excellent experience for enabling GitOps feature flag workflows, with minimal operational overhead. You simply need Flipt and your repository provider to get up and running (the relational backend becomes optional, depending on whether or not you need authentication). This is great for quickly validating Flipt and should scale reasonably well for small to medium-sized projects and teams. The primary bottleneck for scale becomes the size of the source repository storing your flag state.
When configured this way, Flipt needs to clone the entire target repository and keep it in sync. We want Flipt to be suitable for both multi and mono-repositories, however, the larger the source repository, the greater the initial clone becomes and therefore startup time. Additionally, the higher the contribution throughput, the more work Flipt has to do to stay up to date. Flipt has only really begun to scrape the surface with Git, and there are certainly more optimizations we intend to make to scale this backend. However, OCI registries present a neat solution to this scaling problem.
Using an OCI registry and repository for feature flag bundles means the initial fetched payload is kept to only the state that matters to Flipt. In order to support this, Flipt comes with a set of new subcommands (see flipt bundle) for building, pushing, and pulling feature flag state as OCI bundles. The bundle format is designed such that Flipt only needs to refetch namespace contents when it changes from one update to the next. This is down to how we use separate layers for separate namespaces in the OCI bundles themselves.
One further byproduct of this is regarding the scope of access needed by Flipt. In this mode, Flipt only needs read access at runtime to the repository contents itself. This is opposed to the entire contents of your Git repositories. This is a win for the security-conscious folks out there.
Experimentation
Beyond these scale improvements, there are further benefits in the form of arbitrary OCI repository tags. If you’re familiar with tagging a Docker image latest, then you already know what I am talking about. The flipt bundle commands support arbitrary tagging, and Flipt can be configured to source feature flag state from any desired tag you choose.
An intended goal of the Git declarative backend was to unlock branching feature flag state. Now you have the power of Git branches and tags, you can leverage them for experimenting with feature flag changes in different deployment environments. Instead of building the concept of environments into Flipt's own domain models, we would lean into the benefits of a version controlled storage medium to facilitate this kind of validation. Want to experiment with a change to feature flag state in your PR? Then you could automatically deploy Flipt to track your branch in a preview environment. Need to validate a change to your targeting rules in staging first? Then you could have your staging instance of Flipt track a staging branch in Git instead of main.
This last suggestion however has one big downside, while branches in Git are appropriate for more ephemeral proposed changes (e.g. via pull-requests and preview environments), long-lived branches other than main are frowned upon. We’re strong believers in trunk-based development at Flipt, and we fully intend to help you enable it via feature flags. Expecting you to create long-lived branches for environments in hindsight seems unreasonable.
Where this new approach comes in handy, is instead separate folders can be leveraged in your repositories to represent different environments. The flipt bundle command can be employed to create separately tagged bundles for each environment. This has the added benefit that you can see the difference between your environments in a single commit by simply comparing the contents of each directory.
One last potential use case of note is shareable and ephemeral remote bundles using https://ttl.sh. This is an on-demand ephemeral OCI registry provider. With ttl.sh you simply follow their instructions and tag your bundle with the duration of its lifetime (e.g. ttl.sh/b7e858c6-8462-41d7-b526-863ae96db14b:10m for a 10 minute bundle). Then you can share this bundle with anyone and they can flipt bundle pull it locally or configure Flipt to serve from ttl.sh directly.
A Brief Tour
We’re currently working on a guide that will take you through the steps of setting up Flipt with OCI, as well as how to unlock these kinds of long-lived and ephemeral feature-flag environments in your platform. However, here is a quick introduction to get you started.
For this walkthrough, you will need the latest version of Flipt (v1.31.0) installed on your path (head over to our Installation Documentation for more details).
Next, you will need a directory with some Flipt feature state configured in it. For this example, we will create a single file named features.yml. If you want to name your feature flag files something different, check out our documentation on how Flipt locates flag state. The commands we will talk about in this section (flipt bundle build) use the same strategy for locating flag state as Flipt’s other declarative backends (git, local, and object).
# contents of features.yml
namespace: "default"
flags:
- key: "excitingFeature"
name: "Exciting Feature"
type: "BOOLEAN_FLAG_TYPE"
enabled: false
Next, we will create our first bundle based on this state. In the same directory as this new file, run the following:
flipt bundle build mybundle:latest
Given everything works as expected you should see the resulting digest of the bundle printed onto your terminal. Now you have a feature flag bundle created locally. To see all your bundles, run the following:
flipt bundle list
Now that you have built your bundle, we can actually run Flipt over the local bundle itself. This is a great way to quickly validate your flag state before pushing it to some production serving registry. Simply start Flipt like so:
FLIPT_STORAGE_TYPE="oci" \
FLIPT_STORAGE_OCI_REPOSITORY="mybundle:latest" \
flipt
This will start Flipt on http://localhost:8080, where you should immediately see your feature flag is available.
To publish this bundle, we can use the Flipt bundle push command. Here we’re using the ephemeral registry ttl.sh to get going quickly:
export UUID="$(uuidgen | tr '[:upper:]' '[:lower:]')"
flipt push mybundle:latest ttl.sh/$UUID:10m
echo "ttl.sh/$UUID:10m"
This should push your local bundle to the remote registry and print the target repository which can then be used again when running Flipt. Simply take the command to start Flipt earlier and replace the local bundle with the remote one.
# you can manually substitute the repository with the one echoed in
# the previous command if you're in a fresh terminal
FLIPT_STORAGE_TYPE="oci" FLIPT_STORAGE_OCI_REPOSITORY="ttl.sh/$UUID:10m"; flipt
Navigating back to Flipt in your browser should again show your feature flag as it was before. Only this time, Flipt will have used the remote repository to get its state. Try changing your feature flag from enabled: false to enabled: true and then re-build and re-push to the same repository hosted on ttl.sh. Flipt should eventually update (polling interval defaults to 30 seconds) to reflect this change automatically.
What’s next?
This new backend in Flipt is in its infancy, and we have a number of ideas to make the experience of leveraging it easier. The following is a short list of features we’re considering:
More local flipt bundle management commands
For example, commands to help maintain and clean up old bundles or introspect into their contents. Perhaps a little TUI for exploring bundle contents?
A GitHub action to simplify CI integration
A quick and easy GitHub action to drop into your workflows for building and pushing.
Feature flag state attestation
Now that your flag state is in an OCI artifact, you could use a tool like cosign to sign your feature flag state.
Garbage collection
It will be no surprise that at some point someone will want to clear outdated bundles. We think Flipt could eventually help with that too.
We’re excited to see how this resonates with the community and what it unlocks for feature management. OCI is proving to be an invaluable distribution mechanism for all sorts of configuration concerns. We’re just getting started with our own ideas around how to leverage them for Flipt and for broader configuration concerns.
We would love to hear from you about your feature and configuration needs, so come hang out with us on our Discord or open an issue or discussion in our repository.
