
Today we are excited to announce [Flipt Cloud](https://docs.flipt.io/cloud/overview), a fully-managed Flipt experience that allows you get up and running with Flipt feature flags in under 2 minutes.

Flipt Cloud is built on top of Flipt's open-source project but with a twist. We don't lock your data away in our own database. Instead, we use a new approach to store your flag data in your own Git repositories and integrate tightly with your GitHub organization.

### How It Works

When you create an account, we prompt you to connect your GitHub organization or personal account. Once connected, we create a new private repository for you in your GitHub account that is completely separate from Flipt's internal repositories. This repository is owned solely by you and is used to store your flag data.

Our GitHub App will sync your flag data into your private repository and keep it up-to-date. You can manage your flags using the Flipt Cloud dashboard and each change will be a new commit that is pushed to your repository.

<BlogImage
  src="/flipt-cloud/overview.svg"
  alt="Flipt Cloud Architecture"
  width={2800}
  height={1800}
/>

We also allow you to create multiple environments for your flags. This is useful if you want to have a separate set of flags for your development environment, staging environment, and production environment. Each environment can optionally be connected to a different Git repository, or you can choose to connect them all to the same repository under different directories.

### Why Did We Build This?

We've heard from many users that they want a fully managed Flipt experience that allows them to focus on building their applications instead of configuring and managing their own Flipt instance. Many of our users also love our Git-backed storage and the ability to manage flags in their own Git repository which allows them to use their existing workflows and tools.

While most developers are familiar with Git and GitHub, we understand that developers are not the only ones who use feature flags. Product managers, designers, and other stakeholders also need to understand and manage feature flags, and often prefer to use a web-based interface instead of the command line or IDE.

Thus we decided to combine the best of both worlds with Flipt Cloud, allowing both developers and non-developers to manage their feature flags in a web-based interface with the same Git-backed storage that they are already familiar with.

### Why Git and GitHub?

We chose to store flag data in Git repositories because we believe it is the best way to manage feature flag data. Git is a mature and battle-tested system that has become the standard for managing source code. Today, Git is also used to manage configuration data for applications, infrastructure as code, and more. We believe that feature flags are a type of configuration data that should be managed alongside the rest of the application's configuration and code.

<BlogImage
  src="/flipt-cloud/git.png"
  alt="Flipt Cloud Git Backed Storage "
  width={2800}
  height={1800}
/>

Git has a few unique properties that make it particularly well-suited for storing feature flag data:

1. **Version Control**: Git provides a way to track changes to your flag data over time. This allows you to see how your flags have changed and evolved over time as well as revert to previous flag configurations if needed.
2. **Branching**: Git allows you to create branches to experiment with new flag configurations without affecting the main production flag data. Once you have tested your flags on a branch, you can merge your changes into the main branch.
3. **Collaboration and Code Reviews**: GitHub and other source code management systems allow you to create pull requests to review and merge your changes into the main branch. This allows you to get feedback from your team and ensure that your changes are working as expected. More on this in a bit.
4. **Data Sovereignty**: By storing flag data in your own Git repositories, you have full control over your data and can use your existing workflows and tools to manage your flags.

#### Managing Flags with Pull Requests

We are big believers in the power of pull requests to manage code changes. Pull requests are a great way to get feedback from your team and ensure that your code changes are in line with your team's standards and best practices.

We also think that pull requests are a great way to manage flag and configuration changes. Several feature flag vendors provide the ability to propose flag changes and require a review process before the changes are deployed to production. This is particularly useful and often required in regulated industries where changes to production systems require a formal review process.

Instead of building our own bespoke review process, we decided to leverage the existing pull request model provided by most Git hosting providers. GitHub, for example, provides the ability to require reviews from specific users or teams before a pull request can be merged, which we think is a great fit for the types of changes that often need to be made to flag data such as enabling/disabling a feature flag or updating a flag's rollout percentage.

Here's a demo of how this works:

<BlogVideo width={640} height={360} video="1014050658" />

1. A user creates a new branch from a 'protected' environment to experiment with new flag configurations.
2. The user makes the changes to the flag data in the branch and verifies that the changes work as expected by evaluating the flag data from their development environment.
3. Once the user has verified that the changes work as expected, they create a pull request to merge their changes into the 'protected' environment.
4. The pull request is then reviewed by a team member and, if approved, the changes are merged into the 'protected' environment.
5. The new flag configuration is then promoted to the 'production' environment and is now available to be served to production traffic.

You can even [customize the commit and pull request messages](https://docs.flipt.io/cloud/guides/configuring-environments#commit-messages-pull-request-titles-and-bodies) to better fit your team's needs.

### Additional Features

Not only does Flipt Cloud provide a fully managed Flipt experience with Git-backed storage, but we also provide a few additional features that make it easier to manage your feature flags for teams of all sizes.

#### Authentication and SSO

Flipt Cloud provides a fully managed authentication and SSO solution. We support email/password, Google and GitHub social login, as well as magic link login for no charge.

Larger teams that require additional security can also opt to use our Single Sign-On (SSO) solution, which allows you to use your existing identity provider to authenticate users.

#### Custom Role-Based Access Control

Not every team member needs to be able to create and manage feature flags in every environment. That's why we have a few predefined roles such as Admin, Editor, and Viewer with sensible defaults but provide the ability to create custom roles to fit your needs.

<BlogImage
  src="/flipt-cloud/roles.png"
  alt="Flipt Cloud RBAC"
  width={2800}
  height={1800}
/>

You can create custom roles and assign them to your team members to control what they can and cannot do in the Flipt Cloud dashboard and your environments.

#### Environment Protection

We understand that not all flag changes are safe to deploy to production. That's why we allow you to mark environments as protected. Using the above mentioned pull request model, you can create a new branch from a protected environment to experiment with new flag configurations.

Once you have verified that the changes work as expected, you can create a pull request to merge your changes into the protected environment.

<BlogImage
  src="/flipt-cloud/protected.png"
  alt="Flipt Cloud Protected Environments"
  width={2800}
  height={1800}
/>

### What's Cooking?

We are actively working on adding new features and improving the existing ones. Here are a few features we are working on:

1. **One Click Flag Rollback**: We are working on a feature that allows you to rollback to a previous flag configuration if needed quickly and easily.
2. **Signed Commits**: We are working on adding the ability to sign commits to ensure the authenticity and integrity of your flag data.
3. **Edge Evaluation**: We are looking at ways to keep flag evaluation close to your users by using our own edge proxy which supports our server-side evaluation SDKs as well as OpenFeature clients. We&apos;re also experimenting with integrating with edge KV stores such as [Vercel Edge Config KV](https://vercel.com/docs/storage/edge-config) and [Cloudflare KV](https://developers.cloudflare.com/kv/concepts/how-kv-works/) to reduce latency and improve performance.

### Get Started

You can create an account and get started with Flipt Cloud in under 2 minutes by heading to [flipt.cloud](https://flipt.cloud) today.

We also have [Getting Started](https://docs.flipt.io/cloud/guides/getting-started) and [Moving to Production](https://docs.flipt.io/cloud/guides/production) guides to help you get from zero to production with Flipt Cloud.

You can view [pricing and available plans](https://flipt.io/pricing) on our website, or get in [touch with us](mailto:support@flipt.io) to discuss enterprise plans and custom requirements.

Finally, we would love to hear your feedback and suggestions as you explore Flipt Cloud. You can create a [community post](https://community.flipt.io/c/flipt-cloud/6) on our community forum or hop into our [Discord server](https://flipt.io/discord) to chat with the maintainers and other Flipt users.

We are looking forward to seeing what you build with Flipt Cloud!


Announcing Flipt Cloud


My goal with this post is to paint a picture of what building a developer centred product has meant to us at Flipt. To illustrate some scenarios drawn on past experiences, which have culminated in our latest endeavours to make feature flags more powerful, performant, adaptable and frankly, just delightful. All while being a little bit sarcastic and borderline waxing poetic.

## Delivering Continuously

When Mark created Flipt back in 2019, his goal was to put an end to the proliferation of home-rolled feature flag solutions he had come into contact with throughout his career. He was tired of having to learn the ins and outs of something new every time. How did you integrate with this thing? Can it target specific user attributes? Can it do proportional rollouts? He wanted something simple to integrate with, that was easy to deploy and operate. Something that would drop nicely into a modern tech stack.

I was working with Mark at the time on a CI/CD SaaS product called Codeship. Truth be told, like a lot of self proclaimed Continuous Integration (CI) _and_ Continuous Delivery (CD) companies at the time, we really only had a CI product that could run your deploy script. CD was the hot new thing and everyone was racing to define what is actually meant. Gene Kim’s The Phoenix Project was on the reading list and the Accelerate book had just dropped.

Build and release processes hadn’t really caught up with the Agile Manifesto(s) that had been rearranging our org charts for the last decade. Developers spent most of their time spinning in their desk chairs waiting for CI to go green, only to then have to gradually deploy that change through a series of environments designed to mimic the last one (or they were handing it off to someone else to do it for them). The goal was to foster confidence that the change wouldn’t implode when it landed in the end users lap.

Accelerate taught us that our slow release pipelines were… slowing us down. The companies that were succeeding were the ones that were evolving the quickest. How does a tech company evolve quickly, you ask? It ships changes more frequently. It (in)validates its assumptions more often. Then, if things go wrong, it can also recover more quickly.

If you skip all the hard parts, the trick to CD is to embrace trunk based development and just ship everything in your main branch to production. Congratulations, your developers are moving fast and breaking things, just like in that Jessie Eisenberg movie. The problem that soon becomes apparent is that you’re **breaking things**. Now the customer experience is in free fall. You might be inclined to introduce some more automated checks, maybe some kind of intermediary space you could deploy into and validate those changes in first?

Of course, that would put you right back where you started. This is where feature flags often enter the mix. Instead of just yeeting any change into production, first you give your feature a name and guard the new code with a humble `if` statement. You pull that code routing decision out into an external system, which is queried continuously. Then gradually you enable your feature flag for different cohorts in your application. Maybe you start by enabling it for yourself, then your internal team, then onto a small proportion of real users and so on. Eventually, because you’re a good developer, when the feature is enabled for everyone you remove the guarding flag altogether (hahaha).

<StarCTA />

How do I unlock continuous delivery? How do I integrate this into my codebase? The initial core features of Flipt were designed with these developer problems in mind. The result was a single binary, with pluggable storage options, with pre-built and code-generated clients for various programming languages.

## Who, What, Where, When and Why?

Imagine you have embraced features flags and now you have that nice warm fuzzy feeling. You’re shipping changes more frequently and doing so without jeopardising your end users experience.

But then again, how do you know that? Specifically, how do you know that the feature _is_ enabled, _who_ it’s enabled for and how your application is behaving when enabled vs disabled?

### Is this thing on?

But you “flicked the switch” already. The flag says enabled for my personal user right there in the dashboard. However, the fact of the matter is, you’re clicking around in your application and you can’t see the new behaviour working as expected.

Have you also got one of those vestigial light switches in your house that doesn’t seem to go anywhere? This feeling is irritatingly familiar.

You start to ask yourself; does this feature just not work? Does the feature flag tool not work? Is it returning `true` for my requests?

<BlogImage
  src="/ship-fast/anakin-ff.jpg"
  alt="Is this feature flag on? Anakin Skywalker meme."
  width={1972}
  height={2248}
></BlogImage>

The only way to rule out the feature flag tool is if you're **measuring** the results of flag evaluations within the context of your applications (where they're referenced).

Assuming our feature flagging tool integrates nicely with our observability stack (perhaps it publishes _telemetry_ via some broadly adopted _open_ standard), we now drill into the state of evaluations for our feature flag name, only to discover that the flag is always returning `false`. How is this possible?

It turns out we made a mistake when evaluating our feature flag. We never passed the callers associated organization into the context. This is what our particular flag has been configured to target, and so it never evaluates to `true`.

How easy was it to miss this during local development? Often our local setup doesn’t have access to a representative feature flagging setup. Particularly if the tool is in some external SaaS platform. Perhaps instead we use some simple override configuration in our local clients to pretend that flag is either on or off. However, this isn’t going to help us identify that we’re passing our context correctly in the first place.

Our feature flagging solution needs to integrate with our observability stack or come baked with an observability solution out of the box. At a minimum we need to understand the distribution of evaluation results on a per flag level. Ideally, we annotate directly into traces and events the result of a flags evaluation for the associated transaction. This way we can understand both what states the flags are exhibiting, and correlate at the individual transaction level how the system behaves when the flag is enabled vs disabled.

Ideally, we can run our feature flagging tool locally, or automatically in our continuous integration environments and ensure that behaviour changes in the expected way as we mutate and change our feature flag configuration.

### You have one triggered incident

You wake up at 3am. _Who is calling at this hour?_ I don’t know anyone in San Francisco. Oh that’s right, my colleague forgot about that holiday they had booked 6 months ago and I agreed (5 hours ago) to cover their on-call. An SLO is failing and latency has increased above an acceptable threshold. You acknowledge the page and reach for the light switch. No, that one doesn’t work, the other one.

You crack open your observability tool(s) to identify where and when latency started to increase. Service X started to get slower a couple of hours ago. If you’re lucky, you can see deployments and configuration change events directly in your monitoring tool. Otherwise, it is off to source control or your CI/CD system to learn what’s what.

You happen to commit configuration to version control, so you can see what was deployed around that time and how it was configured. However, no one merged anything around that time. What am I missing? Nothing changed!

Oh that’s right, feature flags. They’re another kind of configuration that (more often) live by their own set of rules. Turning to the feature flag dashboard, or perhaps a channel in your works communication tool of choice, you try to correlate what, when and who made changes. Sure enough you find out that a colleague in a different timezone enabled a feature flag at the end of their day. The change appeared to behave as expected, so they wrapped up and went home.

You review any documentation around the flag to understand the impact of simply disabling it. At this point, disabling the flag outright might be enough. However, the tool might offer a way to revert to the state it was in before the incident occurred. Maybe the flag was on and working fine for a different cohort. You revert the flag and leave a note for the developer to know why their flag state was altered.

At the very least, we need to understand their present and past states. Sometimes it can be useful to know in advance what someone else intends to change them to (peer review and approvals). There should be space to document and describe _why_ changes are being made, when making them. As well as the ability to roll back time if the need arises. Ideally, we can also publish all this information into the rest of our observability stack for ease of discovery.

### Reproducing the scene of the crime

The fire was put out by disabling the feature flag, but at some point we need to turn it back on again. We’re not just going to give up on the feature after all. However, you notice that simply turning the flag on for yourself does not reproduce the problem. Attempting to recreate the problem locally produces similar results.

You remember that another team is actively working on a change in a service you’re depending on. Could there be another feature flag in the mix? Enabling both flags finally causes the latency observed during the incident.

We often don’t develop features in isolation. The situation can arise that it is the intersection of two flags being enabled, which causes the alerts to start firing. Ideally, you need to be able to go back and look at the broader set of feature flag configurations at the time of the incident, in order to recreate what was and wasn’t enabled, and for which targets. Ideally, we can take snapshots of all configurations (flags and broader) at that time. Then we can experiment with them somewhere in isolation, whether that is locally, in preview environments or production itself.

## Developer first, not Developer first and last

When we think of developers at Flipt, we try to imagine the engineer that builds with a product mindset, that interfaces directly and empathises with the customer, that operates their own code (or others) in production and that paves a golden path for their peers to ship code safely. We also acknowledge these tools are not just for developers alone, and that what we build should empower folks to safely hand over control to other functions in the business, where appropriate (with configurable levels of access). When developers choose the tools they need to get their job done, they have a problem domain in mind and stakeholders they’re going to serve. We strive to account for everyone.

### End-user experience

Developers are building products for all sorts of people at the end of day. In a lot of ways feature flags intrinsically acknowledge this fact by definition. They exist to protect the quality of the users experience, while the product evolves beneath them. That was the core goal during Flipt’s inception, and remains an important foundation.

However, we need to acknowledge the fact that adding feature flags sometimes involves integrating external configuration into critical, load-bearing or sensitive parts of our systems. This means we need to cater for situations where we cannot leak sensitive information, introduce transient failures or add (significant) latency.

For Flipt, that means supporting both server and client-side evaluation. Client-side means evaluation takes places within your own applications, over snapshots of complete feature flag configuration delivered asynchronously to your code. Server-side means introducing an API request (HTTP or gRPC) into your code, but it keeps configuration isolated within Flipt itself. Furthermore, being able to take our open-source, self-hosted solution and deploy it into your own infrastructure is key for maintaining predictable stability, scale and cost.

### In it together

Developers work alongside a whole host of other disciplines, including operators, managers, designers, product and customer success to name a few. This is a team sport.

At its core we want Flipt to be self-explanatory. Our environments need to be observable, reproducible, evolvable and revertible. As an operator, you should be able to easily understand _what_ the desired state of the system is in any given moment (past, present and future). It should be easy to correlate external systems with changes in flag state, and in case of emergency revert back to a stable state.

We have interpreted this to mean features flags should be declared as code, committed to source control. However, as non-engineer, we don’t expect everyone to learn to code and navigate source control. So we also need graphical controls designed for broader access and these need to contribute directly to source control. Operators and developers should be able to attenuate access to these controls, meaning roles and policies can be defined and enforced through the APIs we expose.

## Why are you telling me all this?

My goal here was to give you some context into how we think and what we value at Flipt. To tell some of our story so far, and hint at where we’re going.

So, it probably comes as no surprise that I had another motive to bring you here. Our [new cloud product](https://flipt.io/cloud) is waiting for you. You can request access today and we will help get you started with how we like to do flags. All of what I described here has gone, and continues to go into what we’re building.

We **really** want your feedback. We’ve got lots of ideas on where to go next, but your needs are what ultimately matter to us most. So do connect with us and let us know what you’re missing, or how we can do better.

Also, stay tuned as we’re also going to write more about the specifics in the coming weeks. How we integrate directly with Git and GitHub, how peer review and history will be surfaced in the UI and where we’re going with out-of-the-box analytics.

Come chat live with us in person in our [Discord](https://flipt.io/discord), or start a discussion or issue on our [GitHub](https://github.com/flipt-io), or ask questions in our [Community Forum](https://community.flipt.io).

Peace!


Shipping Fast Without Breaking Things


<BlogImage src="/hybrid-cloud/hybrid-cloud.png" width={800} height={448} />

At Flipt, we believe you shouldn't have to choose between security and flexibility. That's why we're excited to announce Flipt Hybrid Cloud, a new solution that combines the best of both worlds. With Flipt Hybrid Cloud, you can run open-source Flipt on your own infrastructure while still taking advantage of managed security features. This means you get the control and customization you need, without sacrificing the peace of mind that comes with a managed solution.

In this post, we'll take a closer look at Flipt Hybrid Cloud and explore some of the key features that make it a game-changer for organizations of all sizes. But first, let's talk a bit about why we decided to build Flipt Hybrid Cloud in the first place.

## Why Hybrid Cloud?

Traditional SaaS solutions for feature flag management are great for getting up and running quickly, but they often come with limitations. You're forced to trust a third party with your sensitive data, and you're at the mercy of their infrastructure and security practices. This can be a dealbreaker for many organizations, especially those in highly regulated industries or with strict security requirements.

On the other hand, self-hosted solutions give you complete control over your data and infrastructure, but they can be a pain to set up and maintain. You're responsible for everything from configuration, to monitoring, to security, which can be a daunting task for smaller teams or organizations without dedicated DevOps resources. Larger organizations may have the resources to manage a self-hosted solution, but they often find themselves reinventing the wheel when it comes to security and compliance.

Flipt Hybrid Cloud aims to bridge the gap between these two extremes, giving you the best of both worlds. You get the flexibility and control of a self-hosted solution, with the modern enterprise features teams expect from a managed service.

## Key Features

So what exactly does Flipt Hybrid Cloud offer? Here are a few of the key features that set it apart:

### Limitless Control

With Flipt Hybrid Cloud, you can run the Flipt feature flag management platform on your own infrastructure. This means you have complete control over your data and can customize the platform to meet your specific needs. You choose where your data is stored, how it's encrypted, and who has access to it.

You get all of the benefits and transparency of our [open-source solution](https://git.new/flipt), including the ability to audit the codebase and contribute back to the community. And because you're running Flipt on your own infrastructure, you can integrate it with your existing tools and workflows, making it easier to adopt and scale within your organization.

Finally, you don't have to worry about per seat pricing or usage limits. With Flipt Hybrid Cloud, you only pay for the features you want, and you can scale up or down as needed without any hidden fees or surprises.

### Managed Security

Flipt Hybrid Cloud includes a range of [managed security features](https://docs.flipt.io/cloud/features/authentication) to help you protect your data and infrastructure. You get social login with GitHub, Google, and Microsoft **for free** out of the box. This makes it easy to secure your Flipt instance with the same authentication provider you use for other services, reducing the risk of unauthorized access.

On top of that, Flipt Hybrid Cloud allows you to integrate with your existing Single Sign-On (SSO) provider, so you can enforce your organization's security policies and ensure that only authorized users can access your feature flags.

<BlogImage
  src="/hybrid-cloud/sso.png"
  alt="Flipt Hybrid Cloud SSO"
  width={2880}
  height={1800}
  caption="Easily integrate with your existing Single Sign-On (SSO) provider"
/>

On top of best in class authentication, Flipt Hybrid Cloud includes the ability to create custom [role-based access control](https://docs.flipt.io/cloud/features/rbac) policies or use one of our built-in policies. This allows you to define granular permissions for your team members, ensuring that each user has the right level of access to your feature flags and configurations.

There is no limit to the number of users you can invite to your Flipt instance, and you can easily [manage their permissions](https://docs.flipt.io/cloud/features/user-management) from the Flipt dashboard. This makes it easy to collaborate with your team and ensure that everyone has the access they need to get their work done.

<BlogImage
  src="/hybrid-cloud/roles-custom.png"
  alt="Flipt Hybrid Cloud Custom Roles"
  width={2880}
  height={1800}
  caption="Define granular permissions for your team members with custom roles"
/>

Besides being able to manage your team's access, Flipt Hybrid Cloud also includes [hosted audit logs](https://docs.flipt.io/cloud/features/audit-logs), so you can keep track of who is making changes to your feature flags and when. This makes it easy to troubleshoot issues and ensure that your feature flags are being used correctly. You can easily filter and search the audit logs to find the information you need, and you can export them to CSV for further analysis.

<BlogImage
  src="/hybrid-cloud/audit-logs.png"
  alt="Flipt Hybrid Cloud Audit Logs"
  width={2880}
  height={1800}
  caption="Easily track changes to your feature flags with hosted audit logs"
/>

## How Does It Work?

Next we'll cover some of the architecture of Flipt Hybrid Cloud and how it works.

Flipt Hybrid Cloud is built on top of the open-source Flipt platform, which means you get all of the benefits of our battle-tested feature flag management solution. You can use the same SDKs and integrations you're already familiar with, and you can take advantage of our powerful targeting capabilities to deliver the right features to the right users at the right time.

In short, you deploy Flipt instances to your infrastructure, and then link them to the Flipt Hybrid Cloud dashboard. Your team members can then log in to the dashboard to create and manage feature flags, and your applications can query the Flipt instances to determine which features to show to your end users.

<BlogImage
  src="/hybrid-cloud/overview.png"
  alt="Flipt Hybrid Cloud Diagram"
  width={4371}
  height={2370}
  caption="Flipt Hybrid Cloud Overview"
/>

Flipt Hybrid Cloud leverages our own HTTP/3 tunneling technology, [Reverst](https://github.com/flipt-io/reverst), which allows you to securely expose your Flipt instances to the internet without having to worry about managing firewalls or configuring VPNs. You can read more about Reverst in our [previous blog post](/so-we-built-reverst). All traffic between Hybrid Cloud and your Flipt instances is encrypted using industry-standard TLS, so you can rest assured that your data is safe and secure.

<BlogImage
  src="/hybrid-cloud/architecture.svg"
  alt="Flipt Hybrid Cloud Architecture"
  width={2880}
  height={1800}
  caption="Flipt Hybrid Cloud Architecture"
/>

Each Flipt instance or environment get's its own unique URL, which you use to access the Flipt dashboard, where you can create and manage feature flags. This means that your team members can easily switch between different environments without having to log in and out of different accounts, making it easy to collaborate and test changes across different environments.

## Get Started Today

Want to give Flipt Hybrid Cloud a try? You can easily give it a try with a temporary environment. Our [getting started](https://docs.flipt.io/cloud/guides/getting-started) guide will walk you through the process of setting up your first Flipt instance and linking it to the Flipt Hybrid Cloud dashboard in less than a minute.

Don't believe us? Check out our quick start video below:

<BlogVideo width={640} height={360} video="969891041" />

You can also check out our [documentation](https://docs.flipt.io/cloud) for more information on how to get the most out of Flipt Hybrid Cloud.

We're excited to bring Flipt Hybrid Cloud to market and can't wait to see how it helps organizations of all sizes deliver better software faster. If you're interested in learning more about Flipt Hybrid Cloud, you can [sign up for a free trial](https://flipt.io/cloud) today. We offer a range of pricing plans to fit your needs, and our team is always available to help you get up and running.


The Future is Hybrid


<BlogImage src="/authz-opa/thumb.png" width={1600} height={900} />

One of the most requested features for Flipt has been the ability to secure the API and UI with authorization. We are excited to announce that as of [v1.43.0](https://features.flipt.io/changelog), Flipt now supports authorization using [Open Policy Agent](https://www.openpolicyagent.org/)!

This feature allows you to define fine-grained access control policies for your Flipt instance. With Open Policy Agent (OPA), you can define rules that determine who can access what resources and under what conditions.

In this post, I'll discuss why we chose OPA, how we went about implementing it, how to configure it with Flipt, and provide some examples of how you can use it to secure your Flipt instances. Finally, I'll discuss some of the future plans we have for this feature.

## Authorization vs Authentication

<BlogImage
  src="/authz-opa/auth.gif"
  alt="Who is your daddy and what does he do?"
  width={480}
  height={258}
  caption="Arnold gets it."
/>

Before we dive into the details of how to configure OPA with Flipt, let's first clarify the difference between authorization and authentication.

Authentication is the process of verifying the identity of a user or system. It answers the question "Who are you?" and is typically done using a username and password, API key, or some other form of credentials.

In Flipt, we support several authentication methods, including [token-based authentication](https://docs.flipt.io/configuration/authentication#static-token), [GitHub OAuth](https://docs.flipt.io/configuration/authentication#github), [OpenID Connect](https://docs.flipt.io/configuration/authentication#oidc), [JWT](https://docs.flipt.io/configuration/authentication#json-web-token), and even [Kubernetes service account tokens](https://docs.flipt.io/configuration/authentication#kubernetes). These methods allow you to secure your Flipt instance and ensure that only authenticated users can access it.

Authorization, on the other hand, is the process of determining what a user or system is allowed to do. It answers the question "Can you perform this action?" and is typically done using access control policies.

Authorization is especially important in large organizations where different teams or departments may have different levels of access to resources. For example, one common use case we hear from our users is that they want to restrict access to certain [namespaces](https://docs.flipt.io/concepts#namespaces) depending on the user's role or team membership.

This is where Open Policy Agent comes in.

## Why Open Policy Agent?

When we set out to add support for authorization in Flipt, we evaluated several different options, including building our own custom solution. However, we quickly realized that most options didn't fulfill all of our requirements, and building a custom solution would be time-consuming and error-prone.

One of the reasons it has taken us so long to support authorization in Flipt is because of the sheer number of options available, and the importance of getting it right the first time. We wanted to make sure we chose the right solution that would be easy enough to use and configure, and was flexible, scalable, and of course secure.

Other than the above mentioned requirements, an additional goal was to make sure that whatever solution we chose would not force any external dependencies on our users. We wanted to make sure that our users could easily integrate Flipt with their existing infrastructure and tools without having to make any major changes such as requiring them to run a separate authorization service.

While OPA [can run](https://www.openpolicyagent.org/docs/latest/integration/) as a seperate service/sidecar, it can also be embedded directly into a Go application as a library, which is what we chose to do with Flipt. This means that you can run Flipt with OPA without having to run a separate OPA server, which simplifies the deployment and configuration process. It also means that developers can test their code integrated with Flipt and authorization policies locally.

Finally, we chose OPA because it is a mature, battle-tested solution that is widely used in the industry. It has a large and active community, which means that there are plenty of resources available to help you get started and troubleshoot any issues you may encounter. OPA integrates well into the Cloud Native ecosystem and has integrations with many other projects, such as Kubernetes, Istio, and Envoy.

In short, OPA checked all of our boxes, and shared many of the same values that we hold dear at Flipt, such as being:

- Easy to use and configure
- Flexible and extensible
- Scalable and secure
- No external dependencies by default
- Cloud Native
- Support declarative workflows / configuration as code

## Implementing OPA in Flipt

Implementing OPA in Flipt was a relatively straightforward process. As mentioned earlier, we chose to embed OPA directly into the Flipt codebase, leveraging the [OPA Go library](https://pkg.go.dev/github.com/open-policy-agent/opa/rego).

While the pull request that added this feature was [quite large](https://github.com/flipt-io/flipt/pull/3008/), the core functionality can be broken down into a few key components:

- Loading and parsing authorization policies and external data
- Creating the evaluation 'engine'
- Mapping requests to 'namespaces', 'resources', 'verbs', and 'subjects'
- Mapping authenticated request data to 'claims'
- Enforcing the authorization policies using middleware

The core of the implementation is a [new GRPC middleware](https://github.com/flipt-io/flipt/pull/3008/files#diff-79c0a66bec87158a91911c892fb84b5c3d00812af71207c90eed70a152c55460) that was added to the Flipt API server. This middleware intercepts incoming requests, extracts the relevant data from the request and the authenticated user (or system), and then evaluates the authorization policies using OPA. Since we are also using [GRPC Gateway](https://grpc-ecosystem.github.io/grpc-gateway/), we get the added benefit of being able to secure both the REST API and the UI using the same middleware.

The majority of the functionality required to load and parse authorization policies and data were integrated using OPA's pluggable [storage](https://pkg.go.dev/github.com/open-policy-agent/opa/rego#example-Rego.Eval-Storage) model. This allows us to easily add support for additional policy backends in the future.

We also tied together our existing [auditing functionality](https://docs.flipt.io/configuration/auditing/overview) with the authorization middleware by ensuring that all request types specify metadata such as the request namespace, resource, and verb. This ensures that the authorization policies have all the information they need to make access control decisions and that we can log all events for auditing purposes.

## Configuring OPA with Flipt

Again, we wanted to make it as easy as possible for our users to configure authorization, so we added a new section to the Flipt [configuration](https://docs.flipt.io/configuration/overview) that allows you to specify the location of your authorization policies and external data.

First you need to define your authorization policies in a separate file. Here's an example of a simple policy that ensures all authenticated users can only read flags in the default namespace:

```
package flipt.authz.v1

import rego.v1

default allow := false

allow if {
    input.request.namespace == "default"
    input.request.resource == "flag"
    input.request.verb == "read"
}
```

Then, you need to tell Flipt where to find your policy file. You can do this via our configuration:

```yaml config.yaml
authorization:
  required: true
  policy:
    backend: local
    local:
      path: "policies/authz.rego"

# required while authorization is still experimental
experimental:
  authorization:
    enabled: true
```

Currently we only support local policy files, but we plan to add support for other backends in the future, such as Git repositories, Cloud Storage, and more.

## Policies

OPA policies are written in a language called [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/), which is a declarative language that is designed to be easy to read and write. Policies are evaluated based on the input data that is provided to them, and Flipt expects these policies to return a boolean value that determines whether the request is allowed or denied.

Policies can be as simple or as complex as you need them to be. You can define rules based on any combination of input data, such as the user's role, team membership, IP address, or any other attribute that you want to use to make access control decisions.

When integrating OPA into Flipt we made sure to provide a rich set of input data that you can use in your policies. This includes information about the authentication, the request being made (e.g. the resource being accessed, the verb being used), and any other context that you may need.

For full details on our input data schema, check out our [Authorization Policies](https://docs.flipt.io/authorization/overview#policies) documentation.

The `authentication.metadata.claims` field is particularly useful when integrating with OAuth or OpenID Connect providers, as it allows you to access any custom claims that are included in the authentication token. This can be useful for making access control decisions based on role or team membership. Many identity providers include standard claims like `sub` (subject), `email`, `name`, and `groups`, but can also include custom claims like `role` or `team` if you configure them to do so.

In fact, we created a guide on how to configure Flipt, OPA, and Keycloak to work together to provide a complete end-to-end solution for [securing your Flipt instance with Role Based Access Control](https://docs.flipt.io/guides/operation/authorization/rbac-with-keycloak).

For help writing and debugging policies, OPA provides a [Rego Playground](https://play.openpolicyagent.org/) where you can write and test your policies in a web-based environment. This can be a great way to experiment with different policies and see how they behave before deploying them to your Flipt instance.

There is also a linter available for Rego called [Regal](https://docs.styra.com/regal) that can help you catch common mistakes and enforce best practices when writing policies.

## Using External Data

In addition to the input data that is provided to the policy engine, you can also use external data sources in your policies. This allows you to make access control decisions based on data that is stored outside of Flipt or your identity provider(s).

For example, you could use an external data source to store a list of users who are allowed to access certain resources, or a list of IP addresses that are allowed to access your API. You can then query this external data source from your policy to make access control decisions.

Just as with policies, Flipt supports loading external data from a local file. Here's an example of how you can configure Flipt to load external data from a local file:

```json policies/data.json
{
  "admins": ["mark@flipt.io", "george@flipt.io"]
}
```

```yaml config.yaml
authorization:
  required: true
  policy:
    backend: local
    local:
      path: "policies/authz.rego"
  data:
    backend: local
    local:
      path: "policies/data.json"
```

The accompanying policy could then use this data to make access control decisions:

```
package flipt.authz.v1

import data

default allow := false

allow if {
    input.request.namespace == "default"
    input.request.resource == "flag"
    input.request.verb == "delete"
    input.authentication.metadata["io.flipt.auth.email"] in data.admins
}
```

External data is particularly useful if you need to make access control decisions based on data that changes occasionally, such as a list of employees, but do not need a full fledged identity provider. By storing this data in an external file, you can update it without having to modify your policies.

Flipt supports the ability to refresh external data (and policies) on a regular interval, so you can keep your authorization model up to date without having to restart your Flipt instance.

## What's Next?

We are excited to see how our users will use this new functionality to secure their Flipt instances, and we're looking forward to hearing your feedback on how we can improve it.

> We believe that you shouldn't have to pay extra for common security features, which is why we are adding this feature to Flipt Open Source.

We know that security is important to our users, and we want to make it as easy as possible for you to secure your Flipt instances.

As mentioned earlier, this feature is still [experimental](https://docs.flipt.io/configuration/experimental) and we plan to make several improvements in future releases. Some of the _smaller_ features we are planning include:

- Support for additional policy backends, such as Git repositories, Cloud Storage, and more
- Support for additional input data source backends

On top of that, we're actively working on building Role Based Access Control policy configuration and management directly into [Flipt Hybrid Cloud](https://docs.flipt.io/cloud/overview), so you don't have to write policies to secure Flipt. More on this in a future post.

If you're interested in being an early adopter of this feature, sign up today at [https://flipt.io/cloud](https://flipt.io/cloud) and give it a try. We're looking for users who are interested in securing their Flipt instances with managed Authentication and Role Based Access Control, and we'd love to hear from you.


Authorization With Open Policy Agent


<BlogImage
  src="/reverse-tunnel/thumb.png"
  alt="Revesrt: Reverse Tunnel in Go"
  width={1456}
  height={816}
/>

We recently open-sourced [Reverst](https://github.com/flipt-io/reverst), a reverse tunnel implementation built on QUIC and HTTP/3.

This new project sits at the heart of Flipt Hybrid Cloud to provide efficient, scalable, and secure connections from our cloud to your instances of Flipt.

This blog post aims to explain a little bit of the why (or rather asks _why not?_) and a lot more on the how, specifically the how in Go.

## Building Flipt Cloud

If you don't already know, we build Flipt, the open-source, Cloud Native feature management solution.
Flipt is designed to be self-hosted on your infrastructure.
It is simple to deploy and scale on most modern containerized platforms to serve your applications and workloads.
We believe you get the most benefits by running it yourself, and we want you to keep doing that.

This past week we announced our new [Hybrid Cloud](https://docs.flipt.io/cloud/overview) offering for the Flipt ecosystem.
We say hybrid because we want you to keep running Flipt for feature flag evaluations on your infrastructure.
Self-hosting flags has so many benefits, including lower costs, improved stability and performance as well as security.
However, we believe can make all the other non-functional requirements for managing your Flipt a lot easier by integrating with our Hybrid Cloud offering.

Flipt exposes an API and a user interface that consumes it.
We've built our platform to safely expose this API and in particular the UI on the public internet, all without the need to manage DNS, TLS, load-balancers, OAuth, or session credentials.
We've simplified the process down to a few clicks and an API key.
This means you can get right down to deploying and scaling Flipt while organizing your teams' access in our remote platform.

To make it simple, we built our system around reverse tunnel technology.

## Don't call us, we'll call you

There are _a lot_ of commercial and open-source projects doing reverse tunneling.
Admittedly, we **did not** do our homework well enough before setting out on this adventure.
We evaluated two managed, predominantly commercial (they have open-source components) offerings: Ngrok and [Inlets](https://inlets.dev/).
Both are amazing and you should definitely read more about them, and probably use them if you want to leverage tunnels for your own things.

However, blinded by that inner feeling of "how hard can this be?" and a shallow understanding of a shiny (not that) new thing (HTTP/3 and QUIC) we built Reverst instead.
Why in Go, you ask? It’s all I know, stop asking silly questions, I’m too dumb for Rust.
Why HTTP/3 and QUIC? I’ll get to that later.

But in all seriousness, Go is actually pretty great for this, and mostly in part to the [quic-go](http://github.com/quic-go/quic-go) project. I want to give a massive shoutout to the `quic-go` folks for leading the charge with QUIC for Go. I would probably have used something off the shelf and saved everyone from Reverst and this blog post otherwise. Thank you for the awesome hard work, for being a welcoming place to contribute back to, and for enabling us to build silly things.

## Establishing the Tunnel

In this section, we’re going to _Go_ (not sorry) deep at some of the inner workings of Reverst. There will be lots of Go snippets, you have been warned.

<BlogImage
  src="/reverse-tunnel/reverst_overview.png"
  alt="Reverst Overview"
  width={2383}
  height={1888}
  caption="Overview of how Reverst works internally"
/>

Reverst is built around the same core principles as most other reverse tunnels.
Instead of building a service that listens and accepts connections within its own deployment environment, have it dial out to an instance of a reverse tunnel (i.e. Reverst in this case). Then the remote tunnel server authenticates the request, registers the connection and each party switches their roles from server to client and vice-versa.

> 💡 This is why Ngrok is so great for local development. Services running in hard-to-reach networks (e.g. behind NATs like your home network or on mobile devices) can establish themselves on tunnels deployed in reachable network locations.

In Reverst, a client wishing to establish a listening tunnel opens a QUIC connection to a Reverst server, then starts an initial stream and writes a `RegisterListenerRequest`. The purpose of this request is twofold:

1. Authenticate the caller with Reverst
2. Identify a target _tunnel group_

The following Go code is adapted directly from our client listener code, which is attempting to register itself on a remote Reverst instance:

```go
stream, err := conn.OpenStream()
if err != nil {
	return fmt.Errorf("accepting stream: %w", err)
}

defer stream.Close()

enc := protocol.NewEncoder[protocol.RegisterListenerRequest](stream)
defer enc.Close()

req := &protocol.RegisterListenerRequest{
	Version:     protocol.Version,
	TunnelGroup: s.TunnelGroup,
}

auth := defaultAuthenticator
if s.Authenticator != nil {
	auth = s.Authenticator
}

if err := auth.Authenticate(stream.Context(), req); err != nil {
	return fmt.Errorf("registering new connection: %w", err)
}

if err := enc.Encode(req); err != nil {
	return fmt.Errorf("encoding register listener request: %w", err)
}
```

Tunnel groups sit at the heart of Reverst's ability to load-balance multiple proxied requests across a number of tunneled listeners. They’re primarily the _name_ that associates these connections together. Each named tunnel group has a set of additional configurations including which authentication mechanisms and credentials it has enabled, along with which hostnames the proxy API should route to the group. When an API request that needs proxying is made to Reverst, it uses the request’s hostname to identify the tunnel group and route it to a relevant connection in the pool.

<BlogImage
  src="/reverse-tunnel/reverst_tunnel_group.png"
  alt="Reverst Tunnel Groups"
  width={954}
  height={610}
  caption="Overview of a single Reverst tunnel group"
/>

In order for Reverst to know which tunnel groups it can serve, which hostnames to route to each group, and which authentication mechanisms and credentials to use, it needs this information encoded somewhere. Configuring these parameters is achieved through a YAML configuration file, which looks much like the following:

```yaml
groups:
  "a": # tunnel group name
    hosts:
      - "a.host.com" # hostname for routing proxied HTTP requests
    authentication:
      basic:
        username: "user"
        password: "pass"
  "b":
    hosts:
      - "b.host.com"
    authentication:
      bearer:
        token: "somesecretkey"
```

Reverst can source this configuration from either the file system or directly from a [Kubernetes](https://kubernetes.io/) ConfigMap. It can be configured to watch either of these sources and update its tunnel group and routing details without needing to restart the process. You won’t need three guesses to figure out which popular container orchestrator we’re using to deploy our instances of Reverst.

When a Reverst server instance receives a new connection it attempts to:

- Accept an initial stream
- Read and decode a `RegisterListenerRequest`
- Identify the tunnel group is one that is known to Reverst
- Authenticate the request using the tunnel groups’ authentication credentials

The next Go snippet is adapted from the code in Reverst which handles this process:

```go
stream, err := conn.AcceptStream(conn.Context())
if err != nil {
	return fmt.Errorf("accepting stream: %w", err)
}

defer stream.Close()

dec := protocol.NewDecoder[protocol.RegisterListenerRequest](stream)
defer dec.Close()

req, err := dec.Decode()
if err != nil {
	return fmt.Errorf("decoding register listener request: %w", err)
}

w := &responseWriter{enc: protocol.NewEncoder[protocol.RegisterListenerResponse](stream)}
defer func() {
	// .. cleanup resources and emit telemetry
}()

// tripper here is the name we have for our connection pool
// it's weird I know, but it relates to the fact we handle
// our connections as a set of http.RoundTripper instances
tripper, ok := s.trippers[req.TunnelGroup]
if !ok {
	// respond with an appropriate error
	return err
}

if err := s.handler.Authenticate(&req); err != nil {
	// respond with an appropriate error
	return err
}

// encodes a RegisterListenerResponse with CodeOK
if err := w.write(nil, protocol.CodeOK); err != nil {
	return fmt.Errorf("encoding register listener response: %w", err)
}

// add the connection to the pool
tripper.register(conn)
```

You can see at the end of this code block, Reverst encodes a `RegisterListenerResponse` type back onto the stream, which is then finally closed. This is when and where the connection associated is added to a “pool” of connections. To understand better how the pool is modeled in Reverst, it helps to reflect on how Go’s standard `http.Client` works, as Reverst uses these to perform the proxying of requests.

```go
type Client struct {
	// Transport specifies the mechanism by which individual
	// HTTP requests are made.
	// If nil, DefaultTransport is used.
	Transport RoundTripper
	// ...
```

The Go `net/http` client is a wrapper around the interface `http.RoundTripper`. A round-tripper is the thing that actually performs the writing of the request and reading of the response onto some underlying connection(s). The higher-level `http.Client` handles details of HTTP such as the normalization of certain headers, cookies, and redirects.

The `quic-go` project uses the `http.RoundTripper` interface as its point of integration for supporting `HTTP/3`. Their sub-package `http3` exports its own implementation [http3#RoundTripper](https://pkg.go.dev/github.com/quic-go/quic-go/http3#RoundTripper). Reverst goes a step further and implements its own wrapper around the `http3.RoundTripper`. This allows it to maintain its own set of connections.

Where do these connections come from? You guessed it, they get added by that line at the end of the registration process:

```go
// add the connection to the pool
tripper.register(conn)
```

The Reverst implementation of `http.RoundTripper` does one last thing, and that is to implement round-robin style load-balancing. This ensures that each proxied inbound request is distributed evenly over the set of open connections that have been registered:

```go
func (r *roundRobbinTripper) RoundTrip(req *http.Request) (*http.Response, error) {
	defer func() { /* some cleanup */ }()

	for {
		// set is an internal type that yields each http3.RoundTripper from the
		// set, in order, on each subsequent calls to Next(...)
		// it supports safe concurrent addition, removal and yielding of entries
		rt, ok, err := r.set.Next(req.Context())
		if err != nil {
			// can only be context error
			continue
		}

		if !ok {
			return nil, net.ErrClosed
		}

		resp, err := rt.RoundTrip(req)
		if err != nil {
			if errors.Is(err, net.ErrClosed) {
				r.set.Remove(rt)
				continue
			}
		}

		return resp, err
	}
}
```

Congratulations, your connection is in the pool, and now it's time for both sides of the connection to switch roles.

```go
// register server as a listener on remote tunnel
if err := s.register(conn); err != nil {
	return err
}

log.Info("Starting server")

return (&http3.Server{Handler: s.Handler}).ServeQUICConn(conn)
```

From here, `quic-go` and their `http3` implementation does all of the hard work.
The client switches to start serving incoming HTTP/3 requests from accepted streams. The Reverst server already has the `RoundTripper` pool embedded inside of a `http.Client` which is stored in a map keyed by the hostnames defined in the tunnel ground configuration.

```go
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	start := time.Now().UTC()

	log := slog.With("method", r.Method, "path", r.URL.Path)
	log.Debug("Handling request")

	// bit of host normalization necessary for deployment behind proxies
	host, _, perr := net.SplitHostPort(r.Host)
	if perr != nil {
		host = r.Host
	}

	if forwarded := r.Header.Get("X-Forwarded-Host"); forwarded != "" {
		host = forwarded
	}

	// get http client with embedded round tripper
	s.mu.RLock()
	client, ok := s.clients[host]
	s.mu.RUnlock()
	if !ok {
		log.Debug("Unexpected client host requested", "host", host)
		http.Error(w, "bad gateway", http.StatusBadGateway)
		return
	}

	// clone the request, force HTTPS and change the target host to that
	// of the tunnels serving host address
	r = r.Clone(r.Context())
	r.URL.Scheme = "https"
	r.URL.Host = s.conf.TunnelAddress
	r.RequestURI = ""

	resp, err := client.Do(r)
```

Here we see the Reverst server’s HTTP handler for proxying requests to the set of `http.Client`'s with their embedded `http.RoundTripper` pools. The calling requests target hostname is used to reference the client, the request is copied and manipulated, then forwarded onto the identified client.

In Flipt Cloud we deploy and configure Reverst so you don’t have to (you don’t even have to know it exists).
We then tightly integrate it with our user management system and API gateway.
This allows us to provide a point-and-click experience for managing SSO providers and configuring your internal organization and its users.

## Why HTTP/3 and QUIC?

Finally, why have we chosen to build on HTTP/3 and QUIC instead of alternatives?
Most other popular tunneling solutions I've come across are built on top of web sockets and TCP.

HTTP/3 is the newest standard for the HTTP protocol family.
The primary differentiator for HTTP/3 is that it is built on top of the QUIC protocol, as opposed to TCP (this is the basis for both HTTP/1 and 2).

HTTP/2 brought a much-needed improvement over the HTTP/1 protocol.
It solved a deficiency in the request/response protocol known as [head-of-line (HOL) blocking](https://en.wikipedia.org/wiki/Head-of-line_blocking).
It multiplexes multiple requests over a single TCP connection (amongst other general improvements).
However, there still exists yet another HOL blocking problem within TCP itself.

TCP handles the reliable, ordered delivery of packets to a destination.
The internet has a habit of losing packets from time to time.
It's TCP's job to identify this situation and re-transmit these lost packets.
It is in this retransmission phase that TCP actually has another HOL-blocking situation.
While in a state of retransmission, the protocol will block other packets being sent until order is restored.

To mitigate this issue in TCP, a different protocol is required.
However, TCP is ubiquitous and the lingua franca for stateful, reliable, bidirectional connections on the internet.
Replacing something like TCP requires making changes to all the networked systems out there serving the internet.
This is where the group behind QUIC chose another strategy; build this new protocol on top of UDP instead (TCP's other widely deployed, less reliable sibling).

QUIC offers secure (TLSv3 required) and persistent connections with bidirectional streaming over the existing internet's UDP fabric.
Delivering the benefits of TCP, but without the HOL blocking problem.

Reverst takes advantage of these new protocols to build persistent connections, which can handle multiple concurrent requests over unreliable networks.

## Wrapping Up

If we haven’t lost you by this point, thanks for sticking it out!

There are actually a lot of details I have omitted (believe it or not), including how we deploy Reverst in Kubernetes, the custom controller which operates it, how we scale it, and route connections based on TLS SNI. Don’t worry, I’ll probably bore you all with that in the next blog post.

We’re not afraid at Flipt to experiment with something new, and we’re really excited to see how this one plays out. You can sign up to [Flipt Hybrid Cloud Beta](https://flipt.io/cloud) and also learn more about it in our [documentation](https://docs.flipt.io/cloud/overview).

If you want to check out the code for Reverst, you can find it on [GitHub](https://github.com/flipt-io/reverst). We’d love to hear your thoughts on it, and if you have any ideas for improvements, please open an issue or a pull request.

If you want to chat with us about it in real time, come join our [Discord](https://flipt.io/discord).


So we built a Reverse Tunnel in Go over HTTP/3 and QUIC


In the dynamic realm of software development, the ability to swiftly adapt and iterate is paramount. Central to this agility are feature flags, enabling teams to toggle features on and off without redeploying code. Yet, the power of feature flags extends beyond mere toggling, they generate invaluable data that can illuminate how features perform and are engaged with by users. This data, when analyzed, can guide more informed decisions, enhancing user experience and system performance.

Recognizing the critical role of analytics in unlocking these insights, we at Flipt are thrilled to introduce a major leap forward: **first-class analytics** for feature flag evaluations, now available in our [v1.37.0](https://github.com/flipt-io/flipt/releases/tag/v1.37.0) release!

<BlogImage
  src="/analytics-with-clickhouse/analytics.png"
  alt="Analytics in Flipt"
  width={2400}
  height={1350}
  caption="New Analytics in Flipt UI"
/>

## The Journey to Analytics in Flipt

Our path to integrating analytics into Flipt was largely shaped by our community. Feedback and feature requests from our users highlighted a growing need for deeper insights into how feature flags impact user interactions and system performance. We envisioned a solution tailored to the unique needs of our open-source community, one that would provide comprehensive analytics without compromising performance or usability.

### Why Analytics Matter

At the heart of every feature flag lies a decision point: will this change benefit our users? Analytics provide the answer, offering a clear view of user engagement, feature performance, and potential issues. By analyzing flag evaluations, teams can make data-driven decisions, enhancing feature rollouts and system stability.

One common issue when adopting feature flags in your development and release process is the lack of visibility into which flags are being evaluated and how often. This can lead to uncertainty about the impact and lifetime of feature flags, such as the inability to know if a flag is safe to be deleted or not. Analytics in Flipt aims to address this challenge by providing insights into flag evaluations, enabling teams to make informed decisions and optimize their feature flag strategies.

## Architecting Analytics: The ClickHouse Advantage

To accommodate the high volume of data generated by flag evaluations, we embarked on a journey to find the perfect storage solution. After extensive research, [ClickHouse](https://clickhouse.com/) emerged as the ideal choice for its robust data warehousing capabilities, OLAP (Online Analytical Processing) support, and exceptional performance with time-series data. Also, the fact that it is open source and has a vibrant community around it made it a perfect fit for Flipt.

ClickHouse's columnar storage model allows for efficient data ingestion and querying, making it perfect for analyzing flag evaluations across multiple dimensions. Its support for complex queries and real-time analytics means we can now offer insights like flag evaluation counts over time, complete with gap filling to ensure data continuity.

The fact that ClickHouse is designed to handle large volumes of data with minimal hardware requirements was a significant factor in our decision. This ensures that Flipt users can leverage analytics without worrying about performance bottlenecks or resource constraints. ClickHouse also has a [cloud hosted](https://clickhouse.com/cloud) version, which makes it easy for users to get started with analytics in Flipt on AWS or GCP without having to manage their own infrastructure.

## Optimizing our ClickHouse Implementation

Our initial approach, while effective, was somewhat naive in its execution. We stored and queried analytical data in a straightforward manner, which, despite leveraging ClickHouse's efficiency, didn't fully harness its potential for lightning-fast analytics.

As we delved deeper into ClickHouse's features and functionality, we discovered the power of [materialized views](https://clickhouse.com/docs/en/guides/developer/cascading-materialized-views)—a feature that would dramatically transform our analytics performance. Materialized views in ClickHouse allowed us to pre-aggregate data, significantly reducing the computational load for common queries.

By refactoring our analytics implementation to utilize materialized views, we achieved a monumental performance boost. Query times plummeted from an average of 2 seconds to an astonishing 0.02 seconds. This optimization significantly enhances the user experience by providing near-instantaneous access to analytics insights.

<BlogImage
  src="/analytics-with-clickhouse/clickhouse-v0.png"
  alt="Pre-optmization ClickHouse Query Performance"
  width={1274}
  height={430}
  caption="Pre-Optimization ClickHouse Query Performance"
/>

After the [optimization](https://github.com/flipt-io/flipt/pull/2778), the query performance improved significantly:

<BlogImage
  src="/analytics-with-clickhouse/clickhouse-v1.png"
  alt="Post-optimization ClickHouse Query Performance"
  width={1270}
  height={370}
  caption="Post-Optimization ClickHouse Query Performance"
/>

**Note**: This optimized aggregation and query will be in the next release of Flipt.

## Configuration and Event Sourcing

Implementing this feature required a thoughtful approach to configuration and data management. We introduced a new configuration section that allows users to specify how and when data is collected, including a flush period for data dumping and the destination for storage.

```yaml
analytics:
  storage:
    clickhouse:
      enabled: true
      url: clickhouse://clickhouse:9000/flipt_analytics
  buffer:
    flush_period: 2m
```

Our event sourcing model, inspired by our work with [audit events](/audit-events), leverages [OpenTelemetry](https://opentelemetry.io/) (OTEL) for efficient data collection. By adopting OTEL's batching and export mechanisms, we ensure reliable data capture without the need for custom processing. This integration not only simplifies data collection but also enhances visibility into flag evaluations through [span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events), which can be invaluable for debugging and performance monitoring.

For more information on how to enable analytics in Flipt, check out our [documentation](https://flipt.io/docs/configuration/analytics).

## Looking Ahead: The Future of Feature Flags in Flipt

As we celebrate the launch of our analytics feature, we're already looking to the future. Our roadmap includes expanding our analytics offerings to include more detailed aggregations, such as histograms and quantile measurements, based on user feedback.

We also plan to look at other solutions for analytic data storage, giving users the flexibility to choose the best solution for their needs.

## Conclusion

The introduction of analytics to Flipt marks a significant milestone in our journey to provide powerful, developer-driven feature flagging tools. We're excited to see how our community leverages these new capabilities to drive innovation and excellence in their software projects.

If you're as excited about these developments as we are, we'd love to hear from you! Reach out via [GitHub](https://github.com/flipt-io/flipt), [Discord](https://flipt.io/discord), or [X](https://twitter.com/flipt_io) to share your thoughts, feedback, or questions.


Elevating Feature Flag Analytics in Flipt


<BlogImage
  src="/contributor-spotlight-roman-dmytrenko/roman.png"
  alt="Celebrating Roman Dmytrenko"
  width={1200}
  height={675}
></BlogImage>

This is a new type of post that we are calling a "Contributor Spotlight". It's a way to highlight the work of our contributors and share their stories with the world. We are excited to kick off this series with [Roman Dmytrenko](https://github.com/erka), a talented engineer and Flipt contributor.

Roman has been working with the Flipt team over the past several months, and has added some amazing features and functionality in the process. He has a passion for building great products and is always looking for ways to improve our platform. Roman is a great collaborator and is always willing to go the extra mile to help out the team.

Here are just a few of the things that Roman has contributed to Flipt:

- Added the ability to delete multiple API Tokens at once: [#2424](https://github.com/flipt-io/flipt/pull/2424)
- Helped us move all of our UI data to Redux: [#2441](https://github.com/flipt-io/flipt/pull/2441)
- Fixed some issues with Monaco editor and ultimately replaced it with a lighter-weight editor called CodeMirror [#2646](https://github.com/flipt-io/flipt/pull/2646)
- Added support for Azure Blob Storage [#2538](https://github.com/flipt-io/flipt/pull/2538) and Google Cloud Storage [#2589](https://github.com/flipt-io/flipt/pull/2589) for our declarative [object storage](https://www.flipt.io/docs/configuration/storage#object) functionality
- Added membership checking for our [GitHub OAuth](https://www.flipt.io/docs/configuration/authentication#allowed-organizations) integration [#2508](https://github.com/flipt-io/flipt/pull/2508)

I could go on and on, but I think you get the point. Roman has been a huge asset to the Flipt team and we are incredibly grateful for all of his hard work.

I recently had the chance to sit down with Roman via [Discord](https://flipt.io/discord) and ask him a few questions about his background, his life outside of software development, his work with Flipt, and his thoughts on the future.

Here's what he had to say:

**Mark:**

Roman, thanks for taking the time to chat with me today. Can you tell us a little bit about your background and hobbies?

**Roman:**

> I am from Ukraine and I like backpacking in the mountains. I did a lot of trails in Carpathian mountains but it faded away during the Covid era. My new hobby is homemade sourdough bread baking. I see it as some kind of meditation. Dealing with sticky dough requires patience and practice. Your results may vary depending on flour, temperature, and other conditions. But usually I have a tasty toast with my morning espresso.

**Mark:**

Could you tell us a bit about how you got into software development and some of the projects you've worked on in the past?

**Roman:**

> I got into software development decades ago. After university I started working as a Java developer in the banking software industry. The area was very regulated and it had many closed source pieces. I could spend a lot of days trying to reproduce the issue and fix it. During those days I realized that open-source software makes things much easier for developers. You learn from the original code and don't waste time fighting with the decompiler.

> Life moved along and I joined a team that focused on developing games for Facebook. I switched many languages and developed in PHP, ActionScript, Java, and Golang. PHP probably was the main language during that time as it was promoted by Facebook. I saw the birth of the apps, their best times and their sunsets. Everything moved fast but I learnt a lot from it including: trunk-based development, experiments and feature flags with simple conditional statements.

> As the popularity of Facebook games went down, I joined a startup. While I worked there I used Golang a lot. It's easy to learn and use. I also like that it compiles to a single binary as it's easier to distribute and keep up with compliances. It's a breeze to have a scratch Docker image with a Go binary only.

**Mark:**

That's really interesting. It sounds like you've had a lot of diverse experiences in your career. How long have you been contributing to open-source projects, and what drew you to Flipt?

**Roman:**

> I haven't contributed a lot to open-source. I did a few commits here and there which helped to improve or resolve issues related to the project I worked on.

> I explored feature flag solutions in the past project and Flipt was one of them. The team made the decision to move with an in-house solution. The number of feature flags was small and could be managed with environment variables. One day it was suggested to check Flipt on Github again. So I went to see if I could contribute something to the project. I discovered that I could help and I also could learn a lot at the same time.

**Mark:**

We're really glad you decided to contribute to Flipt. Your work has been invaluable to the project. May I ask, what keeps you motivated to continue to contribute to open-source projects like Flipt?

**Roman:**

> I always admire how many things the open-source community does for each developer. It changes the way the software is developed, deployed, monitored and analyzed. And I want to give something back. Even fixing small bugs will give the maintainers more time to focus on the next bigger thing and innovate.

**Mark:**

It really is helpful as a maintainer to have contributors like you who are willing to help out anywhere in the stack.

Before we go, may I ask what are you most excited about for the future of Flipt and the open-source community? Any features you'd like to see added to Flipt or help develop?

**Roman:**

> Flipt has many ways to integrate, configure, and run/deploy the service. But it's still hard to see how each flag is performing inside Flipt.
> Usually the flag is active for some time for the experiment. It would be nice to have insights into evaluation. It also will be great to know when the flag is not in use or misused: either the experiment is over or some service calls the flag that doesn't exist.

**Mark:**

I completely agree. You won't have to wait long for that feature. We're currently working on a new analytics dashboard that will provide insights into how your flags are performing. We'll be releasing this in the next week or so, and we'd love to get your feedback on it.

We're excited to continue to work with you and the rest of the community to make Flipt even better. Thanks for taking the time to chat with me today, Roman. We really appreciate all of your hard work and dedication to the project.

## Conclusion

We hope you enjoyed this first edition of our Contributor Spotlight series. We're excited to continue to share the stories of our amazing contributors and the work they are doing to make Flipt even better.

If you're interested in contributing to Flipt, we'd love to have you join us. You can find more information on how to get started on our [GitHub](https://github.com/flipt-io/flipt/blob/main/CONTRIBUTING.md).

Also, if you have any questions or feedback, feel free to reach out to us on [Discord](https://flipt.io/discord). We'd love to hear from you!


Contributor Spotlight: Roman Dmytrenko


As we bid farewell to 2023, it's time to reflect on a year of immense growth, innovation, and community at Flipt. This retrospective dives into the milestones that marked our journey, celebrating the leaps we've made in feature flag management and the vibrant community that's grown alongside us.

## Product Innovations: A Timeline of Features

When I look back on the year, I'm filled with pride at the strides we've made in enhancing Flipt's capabilities. We've introduced a host of new features and enhancements. Here's a timeline of our most significant feature releases in 2023:

**January: Redefining User Experience and Authentication**

- **UI Experience Overhaul**: We kicked off the year by revolutionizing our user interface to offer a more dynamic, responsive, and modern experience. This transition wasn't just a facelift; it represented a foundational enhancement to how users interact with Flipt, prioritizing speed, ease of use, and intuitive design. [Read more](/embedding-react-in-go).
- **Advanced Authentication Options**: Introduced a suite of new authentication options including OpenID Connect (OIDC), significantly broadening our support for various identity providers. This update allowed seamless integration with popular services like Google, GitLab, Auth0, and Okta, ensuring that Flipt fits effortlessly into diverse security environments and meets the complex needs of modern organizations. [New authentication options](/new-look-and-authentication-options).

**February: Enhancing Security and Integration**

- **API Token Authentication**: We introduced API token authentication, a vital feature for users needing secure and straightforward access control for their automated systems and integrations. [Details on enhanced authentication](/enhancing-authentication).
- **Kubernetes Service Account Authentication**: Recognizing the importance of Kubernetes in modern infrastructure, we added support for Kubernetes Service Account authentication. [Explore Kubernetes integration](https://www.flipt.io/docs/configuration/authentication#kubernetes).

**March: Team Growth and SDK Expansion**

- **Welcoming New Talent**: We expanded our team with the addition of Yoofi Quansah as a Senior Software Engineer. [Meet Yoofi](/welcome-yoofi-quansah).
- **New Go SDK**: Released a comprehensive Go SDK designed to support both GRPC and REST, significantly improving the developer experience. [Discover the Go SDK](/generating-the-flipt-go-sdk).

**April: Organizational Enhancements and Language Support**

- **Namespaces Support**: We introduced the concept of namespaces to Flipt, allowing users to organize and manage their flags, segments, and rules more efficiently. This feature was a significant step towards enhancing the organizational capabilities of our platform. [Learn about Namespaces](https://www.flipt.io/docs/concepts#namespaces).
- **Python REST SDK**: Acknowledging the widespread use of Python among developers, we released our Python REST SDK. This addition made it easier for a broader audience to integrate Flipt into their applications. [Explore the Python SDK](https://github.com/flipt-io/flipt-python).

**May: Embracing GitOps and Declarative Features**

- **GitOps Integration**: We revolutionized our approach by introducing support for declarative feature management. This shift allows teams to define and manage their feature flags and configurations as code, embracing a GitOps workflow for superior consistency and control. [Discover GitOps with Flipt](/gitops-announcement).
- **Audit Events Enhancement**: Bolstered our system with Audit Events, providing detailed logs for changes, enhancing transparency and accountability in feature management. [Learn more](/audit-events).

**June: User Experience and Operational Efficiency**

- **Dark Mode**: Introduced Dark Mode to the UI, catering to user preferences for a more comfortable visual experience during different times of the day. [Changelog details](https://features.flipt.io/changelog/649c32d439c7b8f8fe279ddd).
- **CLI Validation Command**: Launched a new `flipt validate` CLI command, allowing users to validate their declarative data before serving it via Flipt. This feature enhances the reliability and accuracy of feature flag deployment, reducing the risk of errors in live environments. [Documentation](https://www.flipt.io/docs/cli/commands/validate).
- **Introduced Flipt Labs**: A dedicated space for demonstrating Flipt's applications in 'real-world' scenarios. This initiative was designed to provide users with tangible, practical examples of how Flipt can be integrated and utilized within various types of projects. [Explore Flipt Labs](/announcing-flipt-labs).

**July: Advancing Feature Flag Management**

- **New Evaluation APIs & UI Experience**: We rolled out a new set of [Evaluation APIs](https://www.flipt.io/docs/reference/overview) and a revamped UI experience, focusing on simplifying the creation and management of boolean flags and their rollout strategies. This update was aimed at making the process of gradually rolling out features more intuitive and manageable. [More on boolean flags and rollouts](/boolean-flags-and-rollouts).
- **S3 Storage Support**: Expanded our storage options by introducing support for storing declarative feature data in S3. This addition provides users with more choices for their storage needs, catering to a wider range of infrastructure preferences and requirements. [Documentation](https://www.flipt.io/docs/configuration/storage#amazon-s3).

**August: Enhanced Flexibility and Security**

- **Advanced Segment Logic**: Added support for ANDing/ORing multiple segments when setting up evaluation rules, offering greater flexibility and precision in feature management. This enhancement allows teams to tailor their feature rollouts with complex, multi-condition rules. [Changelog](https://features.flipt.io/changelog/64dd2aefd81926ef7df2f72e).
- **GitHub Authentication & PKCE Enhancement**: Introduced GitHub as an [authentication option](https://www.flipt.io/docs/configuration/authentication#github) and added PKCE to OIDC authentications, significantly bolstering security and user convenience. These updates ensure a more secure and streamlined login experience. [Changelog](https://features.flipt.io/changelog/64f0a4923f3963cffa181c7f).

**September: Broadening Observability and Engagement**

- **Webhooks for Audit Events**: Implemented [webhook support](https://www.flipt.io/docs/configuration/auditing/webhooks) for audit events, vastly improving the observability of system changes and enhancing integration capabilities with external monitoring tools. This feature allows teams to receive instant notifications and insights, fostering a proactive approach to system management. [Improving Observability](/improving-observability).
- **Real User Metrics with Datadog**: Added support for capturing Real User Metrics (RUM) and sending them to Datadog when using our Node SDK. This integration provides valuable insights into user interactions and system performance, aiding in informed decision-making and enhancing user satisfaction. [Changelog](https://features.flipt.io/changelog/650327a38df21de96c6be2ce).
- **MacOS Local Version**: Launched a MacOS version of Flipt for local development and testing. This release simplifies the process for MacOS users to engage with Flipt in their native environment, promoting wider adoption and ease of use. [Install via Homebrew](https://www.flipt.io/docs/self-hosted/overview#homebrew).

**October: Streamlining Configuration and Enhancing Customization**

- **CLI Config Enhancements**: Introduced new [CLI commands](https://www.flipt.io/docs/cli/overview) under `flipt config` to initialize and edit Flipt configurations directly from the command line. This update streamlines the setup and management process, making it quicker and more intuitive for users to adjust their Flipt environment to their needs. [Changelog](https://features.flipt.io/changelog/651af3078e2b1140d6e46daa).
- **Webhook Templates for Audit Events**: Added support for Webhook templates, allowing users to configure the payload of audit events that get sent to their webhooks. This feature provides greater control over the information received, ensuring that teams get the precise data they need for monitoring and analysis. [Documentation](https://www.flipt.io/docs/configuration/auditing/webhooks#templates).
- **Typesafe Code Generation for TypeScript**: Experimented with generating typesafe code for feature flag evaluations in TypeScript, offering a more robust and error-resistant development experience for users working in TypeScript environments. [Feature Flags as Types](/feature-flags-as-types).
- **Edge Database Experimentation**: Explored support for edge databases like [Turso](https://turso.tech/) and LibSQL, pushing the boundaries of where and how Flipt can be used and paving the way for future innovations in distributed environments. [Experimenting on the Edge](/experimenting-on-the-edge).

**November: Enhancing Granular Control and Supporting OCI**

- **Namespaced Auth Tokens**: Introduced namespaced authentication tokens, providing users with more granular control over access and permissions. This feature enhances security and flexibility, allowing for precise scoping of resources and actions. [Documentation](https://www.flipt.io/docs/authentication/methods#namespaced-tokens).
- **OCI Image Support for Feature Data**: Rolled out the ability to store and serve declarative feature data via [OCI images](https://www.flipt.io/docs/configuration/storage#oci), providing a modern, scalable solution for managing and distributing feature configuration. [Features, Flags, and OCI](/features-flags-and-oci).

**December: Revolutionizing Performance with Client-Side Evaluation SDKs**

- **Client-Side Evaluation SDKs**: Unveiled new [Client-Side evaluation SDKs](https://www.flipt.io/docs/integration/client), marking a major milestone in Flipt's evolution. These SDKs dramatically improve evaluation performance by up to 1000x, offering an unprecedented speed and efficiency boost. This innovation allows for near-instantaneous feature flag evaluations directly within the client environment, significantly enhancing the user experience and system responsiveness. The SDKs are designed with modern applications in mind, ensuring that Flipt can meet the needs of highly dynamic and interactive client-side applications where performance is critical. [New Client-Side Evaluation](/new-client-side-evaluation).
- **Azure Blob Store Support**: Expanded our storage options by introducing support for storing declarative feature data in [Azure Blob Store](https://www.flipt.io/docs/configuration/storage#azure-blob-storage). This addition provides users with more choices for their storage needs, catering to a wider range of infrastructure preferences and requirements. [Changelog](https://features.flipt.io/changelog/659033010be3e19e94a3d02c).

## Community Growth: The Heart of Flipt

2023 was a landmark year not just for our product's capabilities but also for the vibrant community that surrounds Flipt. As we reflect on the past year, it's the growth and engagement of our community that stands out as a true measure of our success. Many of the features above were inspired by the feedback and insights of our users and some even fully developed by our community.

We're proud to have built a community that's not just passionate about our product but also committed to our vision of empowering developers and teams to build better software.

**GitHub Surge**: Close to 3.1k GitHub stars, our GitHub community nearly doubled!

**Discord Growth**: Our [Discord](https://flipt.io/discord) family expanded to 212 members, buzzing with ideas and support.

**Contributor Onboarding**: An influx of active contributors on [GitHub](https://github.com/flipt-io/flipt), bringing fresh perspectives and enthusiasm.

**KubeCon Connections**: Attended KubeCon in Amsterdam, getting to meet the [OpenFeature](https://openfeature.dev/) team and other enthusiasts.

**NYC Retreat**: A December gathering in NYC for roadmap planning and user feedback, strengthening our resolve and vision.

**Empowering Open Source Participation**: We're committed to growing the open-source community, actively engaging and sharing our insights on platforms like [Dev.to](https://dev.to/flipt).

**Collaborating with Other Amazing Organizations**

- **Ensuring Quality Leveraging Dagger**: Our journey to enhance our build, test, and release processes brought us into alignment with technology leaders like [Dagger](https://dagger.io/). Integrating with Dagger's cutting-edge tools is a step forward in not just evolving Flipt but also in contributing to a richer, more collaborative open-source ecosystem.

- **Engaging with Turso**: As we ventured into supporting edge databases, we engaged with innovative platforms like [Turso](https://turso.tech/). This interaction is part of our commitment to embracing forward-thinking technologies that push the boundaries of what's possible with Flipt. We are dedicated to continued exploration and integration with technologies that offer our users enhanced capabilities and performance.

## Forward into 2024 and Beyond

Reflecting on the strides we've made in 2023, I'm filled with gratitude for the vibrant community that has been integral to our journey. Our commitment to open-source remains unwavering. We understand that it's not just about building software but nurturing an ecosystem where everyone contributes, learns, and grows together. We're dedicated to ensuring Flipt remains a collaborative, transparent, and community-driven project.

Looking ahead, our vision is clear: to continue evolving Flipt into the most developer-loved feature management solution. We're not just creating tools; we're crafting experiences that resonate with the needs of high-functioning engineering organizations and platform teams.

We believe that the best is yet to come. As we step into the new year, we're more committed than ever to empowering teams, simplifying complexities, and setting new benchmarks in feature management.

Happy New Year! 🎉

---

Interested in onboarding to our Managed Services? [Sign up for early access](https://www.flipt.io/docs/managed/overview). We're excited to partner with you on your feature management journey.


2023 A Year in Review


At Flipt, we have embarked on a journey to discover new ways of speeding up flag evaluations for our users. The need for this is justifiable as feature flags often guard critical application paths where latency must be kept to a minimum. In previous blog posts, we explored using [Flipt as a sidecar](/flipt-as-a-sidecar), and [experimenting with Flipt on the Edge](/experimenting-on-the-edge). Both of these experiments were successful in reducing flag evaluation latency, and also leveraged notable patterns that are becoming more popular nowadays.

Despite the success of these experiments, there remained one huge dependency, the network. On every evaluation, clients would have to go over the network to some Flipt server to get the data it needs. The difference between using a sidecar or edge database and making evaluations to a central Flipt server is that the server and database were closer to the clients which implicitly reduces latency. However, there are scenarios where the network might not be as readily available or where adding any latency to critical code paths is a non-starter.

These are a few of the reasons why true client-side evaluation has been a [heavily requested](https://github.com/flipt-io/flipt/issues?q=is%3Aissue+is%3Aopen+evaluation+client) feature. The idea is that clients would either poll an upstream server on an interval or be pushed data when new data arrives. When evaluation data is later requested, the clients can source it from the same memory, avoiding additional network hops. The main pro of this approach is that data being accessed from memory will always be faster than going over the network. The downside is the potential lack of data consistency since events are being realized asynchronously.

Knowing this, we have deliberately avoided naively implementing client-side evaluation, as it would require us to write the evaluation logic in every language we want to support. Not only is this a time-consuming effort, but implementation parity between evaluation logic on clients and server in this case is extremely critical, opening up the door to introducing hard-to-track-down bugs. As we explored more options, we encountered a mechanism called [Foreign Function Interface](https://en.wikipedia.org/wiki/Foreign_function_interface) (FFI) which allows a program written in one language to call and make use of functionality written in another. Leveraging this standard, we realized that the evaluation logic could be written once and clients in each native language could make use of FFI to call the evaluation logic. This would allow us to support true client-side evaluation without the overhead of implementing the dense logic over and over again.

The rest of this post is dedicated to the architecture and implementation details of our client-side evaluators using FFI.

## Internal API

Before getting into how the client-side evaluation works, we should first cover how our existing server-side evaluation works as well as how new the evaluator running client-side sources the data required for evaluation.

### Server Side Evaluation

For server-side evaluation we have a set of public REST APIs for evaluation in which the [SDKs](https://www.flipt.io/docs/integration/rest) send evaluation request data to the Flipt server, the server evaluates with data from the data backend, then returns the response to the client SDK.

These existing endpoints are:

- [Boolean Evaluation](https://www.flipt.io/docs/reference/evaluation/boolean-evaluation)
- [Variant Evaluation](https://www.flipt.io/docs/reference/evaluation/variant-evaluation)
- [Batch Evaluation](https://www.flipt.io/docs/reference/evaluation/batch-evaluation)

**Note**: Flipt also supports interfacing via GRPC, so we also publish a set of [GRPC SDKs](https://www.flipt.io/docs/integration/grpc) in various languages. You can also [generate your own](https://www.flipt.io/docs/integration/grpc#generate) in your language of choice.

<BlogImage
  src="/client-side-evaluation/server-side-arch.png"
  alt="Server-side Evaluation Architecture"
  width={1815}
  height={1064}
  caption="Existing Flipt server-side evaluation architecture"
/>

### Client Side Evaluation

To support client-side evaluation we had to rethink our approach to how data is sourced from the Flipt server. For the client-side SDKs to perform evaluation themselves without contacting Flipt for each request, we needed a way to supply them with all of the data they required. Without going into all the nitty-gritty details, this effectively meant us building a snapshot of our data model for flags, segments, rules, etc, and then shipping this serialized data structure to the clients.

Luckily, much of the work of building a snapshot of our data model was mostly already done in previous releases of Flipt, as this was necessary to support our [declarative backends](https://www.flipt.io/docs/configuration/storage#declarative) such as Git, OCI, and Object Store. It only took a bit of work to adapt this and put it behind a new ‘internal’ API so that we could request this snapshot state from the clients.

**Note**: While this API is labeled ‘internal’, it is still public-facing, we just wanted to denote that it’s not ‘officially’ meant for consumption by end users without using one of our new SDKs.

<BlogImage
  src="/client-side-evaluation/client-side-arch.png"
  alt="Client-side Evaluation Architecture"
  width={1998}
  height={1842}
  caption="New Flipt client-side evaluation architecture"
/>

The client-side SDK engine (described below), polls this new internal API endpoint for evaluation data from a specific [namespace](https://www.flipt.io/docs/concepts#namespaces) at a configurable rate and stores this snapshot locally in memory. This approach does mean, however, that the data clients have can become out of date from the true data on the server. For example, if a flag or ruleset changes after a client has just polled, they wouldn’t receive the new updates until next time around. Allowing this polling to be configurable however, should help end-users find the best balance for their applications when it comes to when to request updated data. In the future, we will be heavily investigating supporting streaming to clients as well as caching support such as via ETAG headers to limit the amount of data transferred over the network.

## Rust Core

Rust has [great documentation](https://doc.rust-lang.org/nomicon/ffi.html) on how to construct an FFI enabled library. Combined with this, and our excitement to get our hands dirty with Rust, we opted to write the evaluation logic in it. The transfer of the server-side evaluation logic to Rust was quite a heavy lift, but through a lot of tests, we achieved 100% parity with the server-side logic.

### Engine

The Rust library has an "engine" that takes in options from a client. The options here are the upstream URL of the Flipt server from which to fetch flag state data and an update interval that tells the Rust core how often to fetch fresh data from the server. As new data is fetched, the Rust core normalizes it into data structures that make evaluations easy on the client. Also, since the data fetching happens asynchronously from client evaluations, the engine implements proper locking of shared data to be concurrency-safe.

### C Interface and Dynamic Library

To allow clients to call into the Rust core, an interface needs to be established between the two languages. This is where Rust shines in allowing a library to expose C interfaces to a callee language to make use of. That interface is generated by using the [cbindgen tool](https://crates.io/crates/cbindgen). The C interface uses types native to the C language which means clients would have to adapt the values they want to pass through the FFI layer using the C interface to C types. Thankfully, most languages have libraries to assist in converting their native types into C-compatible types. You can look at an example of these libraries for [Python](https://docs.python.org/3/library/ctypes.html) and [Node](https://github.com/TooTallNate/ref) to name a couple.

The C interface defines functions that the clients can depend on to be implemented by the engine. This is accomplished by compiling the Rust code down to a dynamically linked library which a host language can import and use. Without getting into too much detail, Rust supports this among other [various ways of linkage](https://doc.rust-lang.org/reference/linkage.html), and most host languages the ability to import a dynamically linked library and use it.

### FFI Snags and Considerations

When implementing an FFI Layer and languages calling into it, it's crucial to be mindful of memory management. All languages have their own way of how memory is allocated and reclaimed for use. There exists a lot of literature on this topic, but here we will briefly talk about the Rust core and what we had to do to make it work. In comparison to other popular languages, Rust’s memory management can be viewed as a somewhat complex. Rust tries to make as many compile-time guarantees as possible when it comes to variables allocated throughout the lifetime of a program as there is no garbage collector. This means that the compiler will not normally allow you to do things that would cause memory leaks or dangling pointers.

In our case, we needed to ensure that the allocated engine lives as long as the host program using the client lives. This means we have to sidestep the compile-time guarantees Rust tries to make and enter what Rust calls "unsafe" territory.

Despite the scary connotation, Rust has great support for navigating this, namely with the [Box](https://doc.rust-lang.org/std/boxed/struct.Box.html) abstraction, which forces a value to live on the heap and also forces the programmer to manage its memory manually. You can see in [our implementation](https://github.com/flipt-io/flipt-client-sdks/blob/06ba78d7e328346e0dd7b22dc4e14515120badfc/flipt-engine/src/lib.rs#L190) how we used `Box` and passed back a `void` pointer back through the FFI layer to the client. This pointer is then [deallocated](https://github.com/flipt-io/flipt-client-sdks/blob/06ba78d7e328346e0dd7b22dc4e14515120badfc/flipt-engine/src/lib.rs#L254) when the host program calls this function.

## Host Languages

For our initial launch, we are supporting four languages: **Node, Python, Ruby, and Go**, but we will soon be adding support for additional languages.

Most of these initial languages have native libraries to convert types into C types, which made the client implementations very thin. For example, you can see the [Go implementation](https://github.com/flipt-io/client-sdks/tree/main/flipt-client-go) only spans two files (excluding tests) and less than 200 lines of code!

Each of these clients exposes an API that should be a bit familiar if you have used our other server-side clients.

- `EvaluateVariant`: takes in a flag key, entity ID, and context and returns a variant evaluation
- `EvaluateBoolean`: takes in a flag key, entity ID, and context and returns a boolean evaluation

It depends on the language of how the values are passed, however it should not differ too much from one to the next. There are also other instructions and considerations on the README for each client on how to manage the allocated memory of the engine throughout the lifetime of the program. It all boils down to calling the `destroy_engine` method on the C API which is called behind a native language class method.

## Packaging and Distribution

Packaging of these clients is an area in which we are actively exploring. The nature of these libraries being dynamically linked and hard to embed makes packaging challenging. Currently, there is no published client for any of the languages, but there are instructions on each language README on how to build the client and use it within your application.

Since the dynamic library (engine) is platform-dependent, we need to build it for each platform (ex: x86, ARM, Windows) and include it in each 'native' language library in order to then package and publish consumable SDKs to the various registries. We would love any feedback or thoughts on how to best package the SDKs and shared libraries for each language/platform. If you have any ideas, please do not hesitate to reach out to us on [Github](https://github.com/flipt-io/flipt-client-sdks) or [Discord](https://flipt.io/discord).

## Performance Comparison

The big win for evaluations moving to the client side is the performance gain that comes with it. Wherever the evaluations are happening, server or client-side, each come with different overheads and trade-offs in regards to performance and memory. For client-side evaluation, you are making the sacrifice for better evaluation speed by incurring more memory build-up wherever the client is located. This can be an acceptable trade-off in many use cases.

The graph below shows the millisecond duration of evaluations between our server-side SDK vs. the new client-side solution out of 1000 evaluations. Keep in mind that the y-axis here is of logarithmic scale (base 10).

<BlogImage
  src="/client-side-evaluation/performance.png"
  alt="matplotlib graph of evaluation latency between the server-side SDK and our new client-side SDK"
  width={640}
  height={480}
  caption="matplotlib graph of evaluation latency between the server-side SDK and our new client-side SDK"
/>

The variance between the two lines is significant, which is why we needed the logarithmic scale on the y-axis. Evaluation latency with the server-side SDK against the server ranged from 0-14 milliseconds, while the client-side evaluation latency ranged from **0-0.1 milliseconds**. That's a 1000x improvement in evaluation latency!

## Conclusion

Today, you can check out our newly open-sourced repository [flipt-client-sdks](https://github.com/flipt-io/flipt-client-sdks) and start building and integrating these clients into your applications to try client-side evaluation with Flipt!

As mentioned above, if any of this excites you, especially the packaging part please do not hesitate to reach out to us on [Github](https://github.com/flipt-io), [Discord](https://flipt.io/discord), or [X](https://twitter.com/flipt_io), we would love to hear your feedback and opinions!


New Client-Side Evaluation SDKs


In the most recent release of Flipt ([v1.31.0](https://github.com/flipt-io/flipt/releases/tag/v1.31.0)) we added support for OCI registries as a backend for feature flag data. The new backend follows in the footsteps of our other [declarative](https://www.flipt.io/docs/configuration/storage#declarative) file-based backends (`local`, `git`, and `object`). This blog post talks a bit about OCI, its growing popularity for non-container usecases and why and how you can leverage Flipt to distribute flags as OCI bundles.

## What is OCI?

OCI stands for the Open Container Initiative. Taking words directly from their landing page:

> The **Open Container Initiative** is an open governance
> structure for the express purpose of creating open industry standards
> around container formats and runtimes.

[https://opencontainers.org/](https://opencontainers.org/)

OCI currently declares and maintains three different specifications:

- [Runtime Specification](https://github.com/opencontainers/runtime-spec)
- [Image Specification](https://github.com/opencontainers/image-spec)
- [Distribution Specification](https://github.com/opencontainers/distribution-spec)

Each of these specifications come together to create a standard set of operating procedures for OCI-compliant container runtimes to fetch, isolate, and execute workloads.

It is probably important to clarify first that Flipt is **not** concerned with running container workloads. Flipt is concerned with the distribution specification (and tangentially with the image specification). OCI-compliant registries have proven extensible and valuable beyond just containerized applications. Their distribution specification includes extensibility points, which can be leveraged to store arbitrary contents with different artifact and media types.

These extensibility points make it possible for Flipt to package and publish its feature flag configuration contents via OCI compatible registries. Some OCI-compatible registries include:

- [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) (Fun fact: our own [Mark Phelps](https://www.linkedin.com/in/markphelps1/) was a lead engineer and architect of GHCR at GitHub)
- [DockerHub](https://hub.docker.com/) (the OG)
- [Google Artifact Registry](https://cloud.google.com/artifact-registry)
- [AWS Elastic Container Registry](https://aws.amazon.com/ecr/)
- [Azure Container Registry](https://azure.microsoft.com/en-gb/products/container-registry/)

To ease our integration into the OCI landscape, Flipt has built new capabilities on top of a project called [ORAS](https://oras.land/). To quote what ORAS is from their site:

> ORAS is the de facto tool for working with OCI Artifacts. It treats media types as a critical piece of the puzzle. Container images are never assumed to be the artifact in question. ORAS provides CLI and client libraries to distribute artifacts across OCI-compliant registries.

Big shout out to ORAS for making this possible in Flipt.

## Prior-art

We’re not the first and certainly won’t be the last project to leverage OCI for non-container artifact distribution. OCI has created a standard for registry APIs, which is broadly useful for an extensible set of artifact and media types. The APIs are also designed around content-addressability, which lends itself to some very desirable caching traits and security improvements.

We got the idea to leverage OCI from past colleagues (shoutout [David](https://x.com/rawkode) and [Marko](https://x.com/mmikulicic)) and further validation from a number of modern technologies already leveraging OCI for non-container image purposes. To name a few:

**Helm** - A package manager for Kubernetes

Since v3.8.0 Helm has supported using OCI registries as a distribution medium for Helm charts.

[https://helm.sh/docs/topics/registries/](https://helm.sh/docs/topics/registries/)

**Timoni** - Another package manager for Kubernetes

From inception, Timoni has leveraged OCI registries for distributing its modules.

[https://timoni.sh/bundle-distribution/](https://timoni.sh/bundle-distribution/)

**Acorn Labs** - A platform for running and sharing applications

Acorns are shareable bundles of configuration for containers, services, and everything in between, which are distributed via OCI-compliant registries.

[https://docs.acorn.io/advanced/publishing](https://docs.acorn.io/advanced/publishing)

**Nuon** - A BYOC platform

Nuon leverages OCI registries for distributing their configuration artifacts.

[https://docs.nuon.co/concepts/vendor-cloud#artifacts](https://docs.nuon.co/concepts/vendor-cloud#artifacts)

## Improving Scale

The [Git backend for Flipt](https://www.flipt.io/docs/configuration/storage#git) provides an excellent experience for enabling GitOps feature flag workflows, with minimal operational overhead. You simply need Flipt and your repository provider to get up and running (the relational backend becomes optional, depending on whether or not you need authentication). This is great for quickly validating Flipt and should scale reasonably well for small to medium-sized projects and teams. The primary bottleneck for scale becomes the size of the source repository storing your flag state.

When configured this way, Flipt needs to clone the entire target repository and keep it in sync. We want Flipt to be suitable for both multi and mono-repositories, however, the larger the source repository, the greater the initial clone becomes and therefore startup time. Additionally, the higher the contribution throughput, the more work Flipt has to do to stay up to date. Flipt has only really begun to scrape the surface with Git, and there are certainly more optimizations we intend to make to scale this backend. However, OCI registries present a neat solution to this scaling problem.

<StarCTA />

Using an OCI registry and repository for feature flag bundles means the initial fetched payload is kept to only the state that matters to Flipt. In order to support this, Flipt comes with a set of new subcommands (see `flipt bundle`) for building, pushing, and pulling feature flag state as OCI bundles. The bundle format is designed such that Flipt only needs to refetch _namespace_ contents when it changes from one update to the next. This is down to how we use separate layers for separate namespaces in the OCI bundles themselves.

One further byproduct of this is regarding the scope of access needed by Flipt. In this mode, Flipt only needs read access at runtime to the repository contents itself. This is opposed to the entire contents of your Git repositories. This is a win for the security-conscious folks out there.

## Experimentation

Beyond these scale improvements, there are further benefits in the form of arbitrary OCI repository tags. If you’re familiar with tagging a Docker image `latest`, then you already know what I am talking about. The `flipt bundle` commands support arbitrary tagging, and Flipt can be configured to source feature flag state from any desired tag you choose.

An intended goal of the Git declarative backend was to unlock branching feature flag state. Now you have the power of Git branches and tags, you can leverage them for experimenting with feature flag changes in different deployment environments. Instead of building the concept of environments into Flipt's own domain models, we would lean into the benefits of a version controlled storage medium to facilitate this kind of validation. Want to experiment with a change to feature flag state in your PR? Then you could automatically deploy Flipt to track your branch in a preview environment. Need to validate a change to your targeting rules in staging first? Then you could have your staging instance of Flipt track a `staging` branch in Git instead of `main`.

This last suggestion however has one big downside, while branches in Git are appropriate for more ephemeral proposed changes (e.g. via pull-requests and preview environments), long-lived branches other than `main` are frowned upon. We’re strong believers in [trunk-based development](https://trunkbaseddevelopment.com/) at Flipt, and we fully intend to help you enable it via feature flags. Expecting you to create long-lived branches for environments in hindsight seems unreasonable.

Where this new approach comes in handy, is instead separate folders can be leveraged in your repositories to represent different environments. The `flipt bundle` command can be employed to create separately tagged bundles for each environment. This has the added benefit that you can see the difference between your environments in a single commit by simply comparing the contents of each directory.

<BlogImage
  src="/oci-backend/diagram.png"
  alt="Demonstration of how to use OCI bundles for feature flag transport"
  width={3162}
  height={1543}
  caption="OCI bundles for feature flag transport"
/>

One last potential use case of note is shareable and ephemeral remote bundles using [https://ttl.sh](https://ttl.sh). This is an on-demand ephemeral OCI registry provider. With ttl.sh you simply follow their instructions and tag your bundle with the duration of its lifetime (e.g. `ttl.sh/b7e858c6-8462-41d7-b526-863ae96db14b:10m` for a 10 minute bundle). Then you can share this bundle with anyone and they can `flipt bundle pull` it locally or configure Flipt to serve from ttl.sh directly.

## A Brief Tour

We’re currently working on a guide that will take you through the steps of setting up Flipt with OCI, as well as how to unlock these kinds of long-lived and ephemeral feature-flag environments in your platform. However, here is a quick introduction to get you started.

For this walkthrough, you will need the latest version of Flipt (`v1.31.0`) installed on your path (head over to our [Installation Documentation](https://www.flipt.io/docs/installation) for more details).

Next, you will need a directory with some Flipt feature state configured in it. For this example, we will create a single file named `features.yml`. If you want to name your feature flag files something different, check out our documentation on how [Flipt locates flag state](https://www.flipt.io/docs/configuration/storage#locating-flag-state). The commands we will talk about in this section (`flipt bundle build`) use the same strategy for locating flag state as Flipt’s other declarative backends (`git`, `local`, and `object`).

```yaml
# contents of features.yml
namespace: "default"
flags:
  - key: "excitingFeature"
    name: "Exciting Feature"
    type: "BOOLEAN_FLAG_TYPE"
    enabled: false
```

Next, we will create our first bundle based on this state. In the same directory as this new file, run the following:

```bash
flipt bundle build mybundle:latest
```

Given everything works as expected you should see the resulting digest of the bundle printed onto your terminal. Now you have a feature flag bundle created locally. To see all your bundles, run the following:

```bash
flipt bundle list
```

Now that you have built your bundle, we can actually run Flipt over the local bundle itself. This is a great way to quickly validate your flag state before pushing it to some production serving registry. Simply start Flipt like so:

```bash
FLIPT_STORAGE_TYPE="oci" \
  FLIPT_STORAGE_OCI_REPOSITORY="mybundle:latest" \
  flipt
```

This will start Flipt on `http://localhost:8080`, where you should immediately see your feature flag is available.

To publish this bundle, we can use the Flipt bundle push command. Here we’re using the ephemeral registry [ttl.sh](http://ttl.sh) to get going quickly:

```bash
export UUID="$(uuidgen | tr '[:upper:]' '[:lower:]')"

flipt push mybundle:latest ttl.sh/$UUID:10m

echo "ttl.sh/$UUID:10m"
```

This should push your local bundle to the remote registry and print the target repository which can then be used again when running Flipt. Simply take the command to start Flipt earlier and replace the local bundle with the remote one.

```bash
# you can manually substitute the repository with the one echoed in
# the previous command if you're in a fresh terminal
FLIPT_STORAGE_TYPE="oci" FLIPT_STORAGE_OCI_REPOSITORY="ttl.sh/$UUID:10m"; flipt
```

Navigating back to Flipt in your browser should again show your feature flag as it was before. Only this time, Flipt will have used the remote repository to get its state. Try changing your feature flag from `enabled: false` to `enabled: true` and then re-build and re-push to the same repository hosted on ttl.sh. Flipt should eventually update (polling interval defaults to 30 seconds) to reflect this change automatically.

## What’s next?

This new backend in Flipt is in its infancy, and we have a number of ideas to make the experience of leveraging it easier. The following is a short list of features we’re considering:

**More local `flipt bundle` management commands**

For example, commands to help maintain and clean up old bundles or introspect into their contents. Perhaps a little TUI for exploring bundle contents?

**A GitHub action to simplify CI integration**

A quick and easy GitHub action to drop into your workflows for building and pushing.

**Feature flag state attestation**

Now that your flag state is in an OCI artifact, you could use a tool like [cosign](https://docs.sigstore.dev/signing/quickstart/) to sign your feature flag state.

**Garbage collection**

It will be no surprise that at some point someone will want to clear outdated bundles. We think Flipt could eventually help with that too.

We’re excited to see how this resonates with the community and what it unlocks for feature management. OCI is proving to be an invaluable distribution mechanism for all sorts of configuration concerns. We’re just getting started with our own ideas around how to leverage them for Flipt and for broader configuration concerns.

We would love to hear from you about your feature and configuration needs, so come hang out with us on our [Discord](https://flipt.io/discord) or open an issue or discussion in our [repository](https://github.com/flipt-io/flipt).


Feature Flags and OCI


Edge computing has become a recent phenomenon for organizations to explore as they scale to meet the demands of a worldwide customer base. It is simply defined as hosting either devices or networks closer to the consumer. Implicitly, as this happens, the user can perceive a much better experience due to the interaction speed of the application to the devices and networks the application interfaces with.

For example, you can imagine that a user in the United Kingdom will benefit from having their data stored in the UK instead of anywhere else in the world. In broader terms, geographical distance from a consumer to a device or network can be significant at times and does not come for free.

In terms of feature flags, the ability to leverage edge computing suits its use case fairly well. Feature flags can often guard critical paths of an application where it is beneficial that the latency of flag evaluation is kept as low as possible.

At Flipt, we have been exploring many different ways to achieve this goal of minimizing latency for flag evaluations. We already have features such as [server-side caching](https://www.flipt.io/docs/operations/deployment#caching), hosting data in an embedded database such as SQLite, or serving flag state straight from a [filesystem](https://www.flipt.io/docs/configuration/storage#filesystem) to name a few, so the idea of edge computing fits the paradigm quite nicely.

In recent weeks, we have been exploring running Flipt on the edge, and the rest of this post will detail the experience we had running Flipt in front of a notable edge database provider, [Turso](https://turso.tech/).

## Turso/sqld/libSQL

Before detailing exactly what Turso provides as a service, let us briefly cover the open-source technologies and their functionalities that power Turso, [libSQL](https://github.com/tursodatabase/libsql) and [sqld](https://github.com/tursodatabase/libsql/tree/main/libsql-server).

### libSQL

libSQL is an open contribution fork of SQLite and thus is fully backward compatible with SQLite. You can actually run a SQLite client over a libSQL database instance. Since SQLite is famously [closed off to external contributors](https://www.sqlite.org/copyright.html), the fork (libSQL) was born, which tries to cater to some of the issues or feature requests that the community wants with SQLite. There are many various distinct features that libSQL provides from SQLite such as [randomized ROWID](https://github.com/tursodatabase/libsql/issues/12), [WebAssembly](https://blog.turso.tech/webassembly-functions-for-your-sqlite-compatible-database-7e1ad95a2aa7), and [many more](https://github.com/tursodatabase/libsql#features).

### sqld

[sqld](https://github.com/tursodatabase/libsql/tree/main/libsql-server) is libSQL running as a daemon in “server-mode”. It not only provides an HTTP interface over libSQL, but also replication abilities from one libSQL instance to another. When running sqld, you can declare one libSQL instance as the primary and other instances as replicas of the primary. sqld will then replicate data from the primary to the replicas. This is a very powerful feature that we will explore in more detail later in this post.

The libSQL repository provides a nice [docker-compose example](https://github.com/tursodatabase/libsql/blob/main/docker-compose/docker-compose.yml) of how to configure sqld, and how it works overall. In addition to simple replication, sqld also provides [bottomless replication](https://github.com/tursodatabase/libsql/tree/main/libsql-server#integration-with-s3-bottomless-replication) in which data from the primary is continuously replicated to an AWS S3 bucket or any other S3 compatible storage such as [MinIO](https://min.io).

### Turso

<BlogImage
  src="/edge-turso/turso-logo.svg"
  alt="Turso Logo"
  width={100}
  height={100}
/>

[Turso](https://turso.tech/) is a platform that provides managed libSQL with seamless replication of data to multiple regions around the world via sqld. With Turso, you can spin up a database in any region it supports as the primary, and this will be the instance that will accept all writes. Based on that primary instance, you can spin up several replicas in any other region which will serve as read-only copies and will be synced with the primary over time, providing the natural eventual consistency of data.

As we mentioned above, from the other features we already have implemented in Flipt, eventual consistency has been a constant (caching, polling of data), and is acceptable for most use cases. There are also disproportionately more reads than writes of data for most Flipt use cases, so the “extra hop” as Turso describes it to get a write into the primary is acceptable. Knowing these two characteristics, it was a no-brainer for us to at least explore this route of providing capability of running Flipt over Turso.

<StarCTA />

### Flipt and libSQL

Flipt can already be run in a variety of storage modes with a [slew of databases](https://www.flipt.io/docs/configuration/storage#databases). We support the most popular RDBMS' such as: MySQL, PostgreSQL, CockroachDB, and last but not least SQLite. The already existing support for SQLite made it a relatively simple lift to support Flipt with libSQL. The functionality for this is included in [v1.29.0](https://github.com/flipt-io/flipt/releases/tag/v1.29.0) release! This release has support for not only running Flipt against libSQL in flat-file, embedded mode, but also in server mode with sqld over HTTP.

Although relatively painless, we did run into a couple of snags to get sqld working behind Flipt. The repository explains that currently libSQL clients are [somewhat limited](https://github.com/tursodatabase/libsql#getting-started) in terms of breadth of support. For instance, only the Rust, and Javascript clients are stable and supported while the rest of the languages are all “experimental”.

For us, the problem with the [libsql-client-go](https://github.com/yquansah/libsql-client-go) client was that the `main` branch did not have functionality for scanning boolean types nor recognition of the `TIMESTAMP` data type and conversion functions from that type to a common string. We [forked](https://github.com/yquansah/libsql-client-go) the repository and wrote patches for the missing functionality to get it working for the time being. We hope to contribute more to the project in the future if support for running Flipt with libSQL/Turso becomes more popular.

### Flipt ‘on the edge’ with Turso

Along with the functionality of the `v1.29.0` release also came an [example](https://github.com/flipt-io/flipt/tree/main/examples/database/libsql) of how to run Flipt against libSQL with replication provided by sqld. This example contains two containers of Flipt running against two containers of sqld, one primary and one replica instance.

You can access both Flipt instances and make edits via the UI, and witness the replication happening in real time. Although this works nice locally for a proof of concept, it does not show the true power of edge computing, so we opted to do an experiment with each employee at Flipt running in front of a Turso instance.

As you create database instances on Turso, the service will give you a URL that you can use to access the logical database. Turso defines a logical database as a collection of libSQL databases each with a schema and data and access to a primary instance (the primary instance itself is included in the logical grouping). The URL is then mapped to the closest instance to the user asking for the data upon query. Again, writes are routed to the primary instance by Turso, but reads can be served from any replica.

#### Experimenting

For our internal experimenti, we each ran Flipt locally, using the URL that Turso provides as [configuration](https://www.flipt.io/docs/configuration/storage#remote). We spun up a primary instance in IAD (Washington D.C, US), which was the closest on average for all 3 employees, and replicas in DFW (Dallas, Texas US) and LHR (London, England).

<BlogImage
  src="/edge-turso/turso-rw.png"
  alt="Turso Requests"
  width={1776}
  height={1759}
  caption="Turso Requests"
/>

On each read of the data we encountered correctness of routing, as each employee was routed to the nearest libSQL instance. For latency, we experimented with the default state of having a replica in each region closest to an employee and took some instances away over time to see if we could witness an increase in latency. Sure enough, as we took away the replica in DFW, the employee closest saw an increase in latency, and the same for the employee near London when we took away the replica in LHR.

<BlogImage
  src="/edge-turso/worldmap.png"
  alt="Worldmap of two of our Turso Instances"
  width={1105}
  height={426}
  caption="Worldmap of two of our Turso Instances"
/>

Given Turso's ability to replicate data to any region in the world in seconds, we could have spun up replicas in any region that Turso supports, and we would have seen similar results. This is a very powerful feature that we are excited to explore more in the future.

We're also looking forward to trying out Turso's [embedded replica feature](https://blog.turso.tech/introducing-embedded-replicas-deploy-turso-anywhere-2085aa0dc242) which would allow you to access a Turso/libSQL database via the native [libSQL C binding](https://github.com/tursodatabase/libsql/blob/main/bindings/go/libsql.go) instead of over HTTP with sqld and the [libsql-client-go](https://github.com/libsql/libsql-client-go) client. This would remove the additional latency of handling HTTP requests/responses and the marshalling/unmarshalling of JSON.

## Conclusion

Turso provides an impressive experience for managing distributed SQLite compatible databases and we are excited to see how it evolves in the future.

At Flipt, we've been actively exploring how to make feature flag evaluation as fast as possible. We are very excited about the many opportunities/technologies that can aid in that mission, including running Flipt with your own data on the edge. In the [previous blog post](/flipt-as-a-sidecar), we explored running Flipt as a sidecar which can work well in a Kubernetes environment, and now we have explored running Flipt on top of a managed edge database provider without Kubernetes.

We're continuing to explore alternative ways to run Flipt on the edge, and we're excited to share more about that going forward.

If you are interested in Flipt, edge data/computing, or how to leverage the power of feature flags in your organization, please feel free to reach out to us via [GitHub](https://github.com/flipt-io/flipt), [Discord](https://flipt.io/discord), [X](https://twitter.com/flipt_io), or [Mastodon](https://hachyderm.io/@flipt).

Until next time!


Experimenting on the Edge


As organizations choose to integrate a feature flag solution within their stack, they should consider several core characteristics carefully. Feature flags can be used in various ways, and often guard critical paths of functionality that an organization wants to slowly rollout, or quickly turn off in case of a disaster. This naturally lends itself to an organization asking about the consistency and availability of the feature flag solution and how fast they could get values from feature flag evaluations to their clients that are leveraging it.

At Flipt, we have had several [feature requests](https://github.com/flipt-io/flipt/issues/2122) for client-side evaluation of flags. Currently, Flipt allows only server-side evaluation, with the option of enabling cache to reduce round trips to the database. This works fine for a good amount of use cases that our users, however, client-side evaluation is still a heavily requested feature.

For client-side evaluation to work, a special client would have to be written in each desired language to asynchronously pull flag state from a central server and store the flag state in memory. Evaluations for flags in this case would certainly be faster because it removes network hops to receive an evaluation value on each evaluation request. This could be a great solution for us, however writing a custom client in each language that users may request takes time away from building other necessary features for the core product.

Acknowledging the above facts, we still want to develop a solution that works for us as a company and our users. For these reasons, we have decided to explore using Flipt as a [sidecar](https://www.oreilly.com/library/view/designing-distributed-systems/9781491983638/ch02.html), a very popular design pattern used in distributed systems.

The remainder of this blog post will focus on what the architecture looks like in this new world and how consumers can leverage it.

### Architecture

Before talking about the overall sidecar architecture, however, it is important to note that today Flipt can be run in a variety of configurations for a storage backend. The traditional way to run Flipt is in front of an RDBMS (SQLite, MySQL, Postgres, etc.). By default, if you were to run Flipt locally it runs with a SQLite database 'next to' the Flipt binary. In addition to traditional RDBMS, Flipt can also be run in [file system (FS) mode](https://www.flipt.io/docs/configuration/storage#filesystem). In this mode, Flipt will periodically fetch from the target FS backend and build an in-memory snapshot of all of it's state. This process is asynchronous from any actual evaluation requests as requests are serviced on a live snapshot of state. The snapshot is only replaced with the new one once it has been built entirely.

It turns out that this architecture is perfect for doing evaluations in a sidecar as it removes one of our network hops from the evaluation call path. For reference, the different FS implementations we support are local file system, Git repositories (check out our Git blog post [here](/gitops-announcement)), and AWS S3 buckets thanks to an [awesome contributor](https://github.com/flipt-io/flipt/pull/1900).

<BlogImage
  src="/sidecar/s3-replication.png"
  alt="S3 Replication"
  width={1972}
  height={2248}
></BlogImage>

This same diagram can be found in our [labs repository](https://github.com/flipt-io/labs/tree/main/sidecar#object-store), where we also demonstrate how to run the proposed architecture in a minikube cluster.

In this diagram, you can see that there is a `flipt-master` process which will be the main source of truth for all feature flag data. This `flipt-master` process can be backed by any storage mechanism. The data from `flipt-master` will be replicated into an S3 bucket at a configurable interval via a Kubernetes cron job. Within the Flipt binary, there is an [`export`](https://www.flipt.io/docs/cli/commands/export) subcommand that reads flag state data out of `flipt-master` and writes that data into a file. That subcommand is what is used for the cron task, and the resulting data uploaded into the bucket.

Any sidecar that will be deployed alongside a service will then run in FS read-only mode with the S3 bucket as the storage mechanism. As the service makes evaluation calls it can source the data from either the [gRPC](https://www.flipt.io/docs/integration/grpc) or [REST](https://www.flipt.io/docs/integration/rest) API. These evaluation calls will be served from the sidecar, which will use data in memory from the S3 object store instead of having to get the data from a database over the network.

### Sample Results

If you run the `minikube` example within the labs repository, there is also another custom application that is written in Go and serves HTML content to the client.

The HTML page allows you to make evaluation calls by selecting either the `sidecar` or the `master` process.

As the instructions state in the [README](https://github.com/flipt-io/labs/tree/main/sidecar), you can choose from `flag_001` - `flag_010` to make evaluations as these flags exist in the initial seed data to `flipt-master`.

<BlogImage
  src="/sidecar/latencies.png"
  alt="Flipt Sample Evaluations"
  width={1527}
  height={708}
  caption="Load test results against `sample-app`"
></BlogImage>

The image above was generated doing a short load test using the [vegeta](https://github.com/tsenart/vegeta) tool. It made requests to the `sample-app`'s `/cli` endpoint at a rate of 15 req/s for 10 seconds, and dumped latency measurements to a csv file which was read and plotted using the [matplotlib](https://matplotlib.org/) + [pandas](https://pandas.pydata.org/) Python libraries. As you can see, evaluations against the sidecar were much faster (up to an order of magnitude) than evaluations against `flipt-master`. Keep in mind that this is just `minikube` as well, which is ultimately a VM running on your machine simulating a Kubernetes environment. In a real distributed environment, the difference in latency between the sidecar and `flipt-master` would likely be much more pronounced.

### Where to Go From Here?

At Flipt, we aim to make evaluations of your feature flags as fast as possible. We realize that adding an external dependency such as Flipt has a cost and we want to minimize it. Because of this, we are devising ways to decrease evaluation time for the source as much as possible, and the sidecar approach is one way to make this happen.

However, we are not done just yet. We're experimenting with additional integrations, architectures, and deployment patterns in the coming weeks to aide in operating Flipt in a multitude of real-world environments. Stay tuned and keep an eye on our [deployment docs](https://www.flipt.io/docs/operations/deployment) as well as this blog for updates.

If any of this interests you or you have any questions, please feel free to reach out to us via [GitHub](https://github.com/flipt-io/flipt), [Discord](https://flipt.io/discord), [X](https://x.com/flipt_io), or [Mastodon](https://hachyderm.io/@flipt).


Flipt as a Sidecar


<BlogImage
  src="/typed/typed.png"
  alt="Introducing Typed"
  width={812}
  height={397}
  caption="Typesafe feature flags with Flipt and Typescript"
/>

Feature flags are amazing tools to help you ship your code safely by guarding your end users against incomplete features in production. Many teams use feature flags in this way, to ship code that makes up a larger feature, enabling testing by a group of internal or beta users without having to deploy the code all at once.

This ability to slowly roll out and guard functionality is especially helpful in environments where multiple applications (services) make up the larger end-user experience. Sometimes these services are owned by orthogonal teams, which makes the communication around and planning of feature releases all the more important.

We believe that feature flags give engineers superpowers, allowing them to change application functionality at runtime, without requiring re-deploys. But, as you may have heard, "with great power, comes great responsibility". This ability to modify a running application through the click of a button or external configuration change (see our post on [Flipt for GitOps](/gitops-announcement) ❤️) also introduces risk.

Feature flags introduce a dependency on an external system, which means your application code is no longer guaranteed to work the same in all environments. At Flipt we believe that the power and flexibility gained through the use of feature flags in production is worth the risk they introduce, however, we do think there is more we can do to help mitigate this risk and remove some of the sharp edges.

## Flag Identifiers

In Flipt, flag keys are unique (within a namespace) identifiers that are assigned to each feature flag in order to differentiate them from one another. It is important to choose meaningful and descriptive flag keys that accurately represent the functionality they control. This will make it easier for developers and stakeholders to understand and manage the feature flags within the system.

When defining flag keys, we recommend to follow a consistent naming convention. This can help maintain a clear and organized system of flags, especially when dealing with a large number of flags. For example, using a prefix or a specific format for flag keys can provide better context and improve readability.

Because flag keys are simple strings in most feature flag domains, they are extremely flexible and can contain almost any value. However, as most programmers know, strings are also easily mistyped which can lead to errors. In the case of feature flags, these errors include incorrect or non-existent flags being referenced, leading to runtime exceptions or unexpected behavior.

To mitigate the risk associated with mistyped flag keys, one common approach is to use a constant to define and manage flag keys. This can provide type safety and help catch errors at compile time rather than runtime for compiled languages. Using constants for flag keys, however, does have its downsides. Nothing stops multiple constants being created for the same value, which means there is no guarantee that when searching your code for a specific constant identifier, that you will find all references for that flag key. There is also nothing stopping you from defining a constant with an invalid value in the first place.

Also, using constants does not give the end user flexibility to define their own type which can enforce limiting a field being of a specific set of values.

```typescript
interface EvaluationRequest {
  requestId?: string;
  flagKey: string; // we only want a subset of possible values here
  entityId: string;
  context?: Record<string, string | undefined>;
}

let req: EvaluationRequest = {
  flagKey: "foo", // this is allowed!
  entityId: "1234",
};
```

## Adding Type Safety

Recently, a user of Flipt [proposed adding type safety](https://github.com/flipt-io/flipt-node/issues/6) to our [Node/Typescript SDK](https://github.com/flipt-io/flipt-node). After some back and forth and some learning on our part, we concluded that we could leverage some of the pieces that we already have in Flipt and create a CLI to generate these types.

When it comes to generating code, you first need a source or specification that can be parsed and then used for writing the code. We realized that for our feature flag data we already have this source, our [features.yml](https://www.flipt.io/docs/configuration/storage#defining-flag-state) file format which we use for [imports/exports](https://www.flipt.io/docs/operations/import-export) and to power our GitOps, Local, and Object Storage data backends!

Again, that same amazing user created a prototype/proof of concept CLI tool that parses the above-mentioned `features.yml` and generates `Flag` and `Context` union types for Typescript. We just open-sourced and published this tool that we’re calling [Typed](https://github.com/flipt-io/typed)!

Usage info is in the README, but here’s how you could quickly generate type-safe Typescript code from your features data in Flipt.

1. Export your feature data from Flipt, either via our API or through a direct DB connection using our [export](https://www.flipt.io/docs/operations/import-export#export) command: `flipt export`
2. Run `npx @flipt-io/typed --lang ts --input /path/to/features.yml` to output the generated Typescript code, or pipe it to a file.
3. Import the types and use in your code!

```typescript
import { Flag } from "./types/flipt";

interface EvaluationRequest {
  requestId?: string;
  flagKey: Flag["key"]; // enforces we can only use valid flag keys that exist in our Flipt instance
  entityId: string;
  context?: Record<string, string | undefined>;
}

let invalid: EvaluationRequest = {
  flagKey: "foo", // this will now error!
  entityId: "1234",
};

let req: EvaluationRequest = {
  flagKey: "language", // correct!
  entityId: "1234",
};
```

Here's what the generated types look like:

```typescript
// Generated by @flipt-io/typed
export type Flag = {
  key: "language";
  value?: "es" | "en" | "fr";
};
```

You can see that the generated types are very simple, and only contain the information that is needed to enforce type safety. The `Flag` type contains a [union string type](https://www.typescriptlang.org/docs/handbook/literal-types.html#string-literal-types) for the flag key and value.

Typed also supports the generation of a `Context` type which is the union of all the possible contexts that are defined in the `features.yml` file. This means that the `Context` type will contain all the possible fields that can be used in the `EvaluationRequest` interface.

## Contributing

[Typed](https://github.com/flipt-io/typed) is still very early days, and considered experimental. There are plenty of features/support we want to add and rough DX edges we want to smooth over, but importantly, we wanted to get it out there and see if anyone finds it useful. We also want to support more languages than Typescript in the future, including but not limited to Go, Java, Rust, Python.. pretty much every language that we have an ‘official’ [SDK](https://www.flipt.io/docs/integration/overview) for.

We’d love it if you gave [Typed](https://github.com/flipt-io/typed) a try and helped us out by filing issues, feature or language requests, or even contributing code. No contribution is too small and we’d love it if the open source community helped drive our roadmap.

In future post we’ll discuss another project we’re working on to improve the developer experience of working with feature flags.. so stay tuned!

As always, feel free to reach out on GitHub or come chat in our [Discord](https://flipt.io/discord). We’d love to learn more about your specific workflow when it comes to adding and working with feature flags in your codebases.


Generating Types for Feature Flags


In this post, I will introduce you to a project called Cup that we have been experimenting with. But before we delve into Cup, let me take you through the journey we have
embarked on to reach this point.

For those in a rush, here are some quick takeaways:

- We need better APIs for reading and writing configuration to Git
- You don’t have time to sweat the Git details, as you should be focusing on your user's needs
- Our new project [Cup](https://github.com/flipt-io/cup) is attempting to solve the above problems

## Gitting Going with GitOps

Back in May of this year (2023) we [proposed our new filesystem backend](https://github.com/flipt-io/flipt/discussions/1652) for Flipt. These backend types aimed to enable a native GitOps workflow for feature flags and users of Flipt. Now Git can be configured as the source of truth for your feature flags.

We then proceeded to build and deliver this capability over the coming months. Fast forward to today, Flipt can be deployed in front of your [local directory](https://www.flipt.io/docs/configuration/storage#local), [a Git repository](https://www.flipt.io/docs/configuration/storage#git), or even [AWS S3](https://www.flipt.io/docs/configuration/storage#object) thanks to one of our [amazing contributors](https://github.com/flipt-io/flipt/pull/1900).

The main downside to using Flipt with a filesystem backend in its current form is that the UI enters into a read-only mode. With these backends, the feature flag state management happens through files committed to git, written to directories, or pushed to object storage. This process happens outside of our current UI’s purview.

Our UI is designed to work over Flipt’s existing management APIs. These APIs operate imperatively and are built around storing state in a relational database backend. While there are ways we could engineer these endpoints to write to storage files, add and commit them to Git, and push these changes, the end experience would still likely leave a lot to be desired.

An ideal solution we have imagined would be for the UI to support both Git and proposal mechanisms such as GitHub’s pull-requests, or GitLabs merge-requests. Changes in the UI could be composed into proposals and then made onto these Git providers. The UI would also then present and track the proposals made back to the user.

## Going in Circles

While spiking on some of the pieces required to make a process like this work, I ended up reflecting on a lot of previous components and glue projects I had built with my past team working on a GitOps CD pipeline. I was reminded of all the pain and frustration many engineers came to us with around our configuration repositories.

- Where should the configuration for X live?
- How do I configure an environment variable for service Y?
- What’s needed to define an entirely new service?
- Where’s my recent application change deployed to right now?

To my team, reconfiguring a service's environment variables was a walk in the park. We did it every single day. But for newcomers, it was a real headache learning the code base. So instead we built tools and automation to read and write the configuration for them. Some of the things we ended up building to help with these problems were:

- Automation to update image versions in our repository, as and when new versions become available (continuous delivery all the way from merging an application change).
- Automation to promote image versions from one deployment environment (staging, production, etc.) to the next (progressive delivery).
- Graphical UIs to explain what versions of things were deployed in which environments.
- Templates for bootstrapping new services and new entire clusters.

At the end of the day, our configuration repository was a means to an end. No internal developer or platform team sets out with GitOps as their goal. GitOps is a process and set of tools to enable the end goal. The end goal is to build a platform that allows application developers to self-serve the infrastructure they need to deploy and manage their applications and to track those changes in something versioned and immutable.

<div className="not-prose flex items-center justify-center">
  <Tweet id="1678118806517067776" />
</div>

The automation we built was designed to either make our configuration and Git more accessible, or hide the need to interact with it altogether. This usually involved adding APIs over the top of these repositories to materialize more meaningful insight, or automation to generate pull requests over GitHub with the desired change for the logical request (new service, updated image version, and so on).

Git is ubiquitous these days, and it has a lot of desirable properties out of the box (versioned and immutable). This is why GitOps is so appealing, but is Git the best interface we have to offer? Let’s face it, it is prohibitive to non-developer or less technical audiences. In my experience, failing to provide more meaningful abstractions around these problems to our engineers led to a couple of undesirable outcomes:

1. Engineers work around you to get what they want
2. Or, they fill your inbox with requests for you to do it for them

I want internal developer and platform teams to have to focus on their developer’s needs and for these manual processes to be automated and for these stakeholders to serve themselves.

## How to Bridge the Gap?

We believe that we need to be building more meaningful APIs over Git. APIs that are designed to serve our stakeholder's needs:

- New service creation
- Getting a database
- Adding a feature flag
- Continuous deployment

Ideally, if you’re in one of these teams, you use a tool that helps you focus on these problems and not the necessary but distracting details of Git.

You should only need to focus on the business logic of parsing the current state, and transforming your repository contents from one revision into another, based on some new desired state. The tool should handle branching, adding, committing, pushing, and opening pull or merge requests for you.

<StarCTA />

You’re likely going to have all sorts of configuration formats and languages written into your repositories. This means the tool needs to be very **extensible**. There is no way a single project can support all the possible formats out there, given the nature of what can and is currently being stored in Git configuration repositories (YAML, JSON, Terraform/HCL, Jsonnet, CUE to name a few).

Beyond just automating away fiddly Git details, we’re going to want to consume these APIs with command-line tools and graphical user interfaces. This is how we reach those users that need a lower barrier to entry. They don’t want to interact with just a raw API as much as they don’t want to interact with configuration in a Git repo. But who has time to build these things? Most smaller to medium organizations don’t have the luxury of dedicating engineering time to these kinds of tasks.

Ideally, there will exist some off-the-shelf command line tools and user interfaces that can interact with any project built with our theoretical Git hiding framework or tool.

For this to work, the APIs should ideally present a **declarative** contract to consumers. This will greatly simplify the complexity of more generically applicable clients. Equally, the contract should be **consistent** across the extensible surface area that the tool can support. Finally, the API needs some capacity to be **self-describing** for generic tooling to present a meaningful interface over its extensible contents.

Does any of this seem familiar? The Kubernetes API server does this very well already.

_Shoutout to [Marko Mikulicic](https://github.com/mkmik) for helping to clarify this vision. I brought an
early prototype to him and he suggested that the Kubernetes API server is a
perfect fit for this problem._

## Building Cup

Introducing [Cup](https://github.com/flipt-io/cup).

A few months back we set out to build a Kubernetes-like API server that facilitates reading from and proposing changes back to Git repositories. A tool with a declarative API for retrieving well-structured and consistent API responses. The API reflects the shape of the data you’ve configured inside of it, such that more general-purpose tools can be developed to access its state. Eventually, Cup should get you a lot of the way without needing to build too much yourself, while also giving you the escape hatch to build your solutions around it.

### Consistent

The API is consistent from one resource kind to the next.

Retrieving, searching, updating, and removing should be the same for a service as it is for a logical feature or database. Retrieval, update, and removal require instances of each kind to be uniquely addressable (group, version, kind, namespace, and name). Search and cataloging require a taxonomy on which to organize and facet discovery (groups, namespaces, and labels).

### Declarative

The API expresses resources through a logical representation of _what_ is contained in the repository, as opposed to _how_ it is stored. The _how_ is defined inside [controllers](https://cup.flipt.io/configuration/controllers), where you can extend Cup to suit your layout and format requirements.

Controllers combine the state of the Git repository at a particular reference, with a desired
state of a resource to calculate a difference to be proposed via a pull or merge request on a target SCM (GitHub, GitLab, Gitea, etc.).

### Self-describing

Each resource has a structured schema, which can be requested via the API. This schema can be used by external tools to present meaningful interfaces or enable further automation to be built.

## Where Do We Go from Here?

Cup is very much an experiment and very much in its infancy. We’re looking to see if it excites people, and what people want to start making with it.

### Status Quo

Out of the box Cup comes with a simple configuration-driven controller for organizing the Kubernetes-like API resources as JSON in your repositories. This has limited scope for customizability. However, it works quite well for raw Kubernetes-style configuration.

The following video does a tour of Cup and the template controller configuring an application deployed with ArgoCD:

<div style={{ position: "relative", paddingBottom: "56.25%", height: "0" }}>
  <iframe
    src="https://www.loom.com/embed/a6adca0a455e4d55991d261f081e4934?sid=7120117f-a4ef-4a2a-b9c9-2c78a46ca3a9"
    frameBorder="0"
    allowFullScreen
    style={{
      position: "absolute",
      top: "0",
      left: "0",
      width: "100%",
      height: "100%",
    }}
  ></iframe>
</div>

Additionally, Cup comes with a very extensible WASM-type controller thanks to the awesome folks building [Wazero](https://github.com/tetratelabs/wazero/) for Go. This controller takes programs compiled to WASM and WASI snapshot preview 1, with a documented command line protocol, and turns them into a controller engine for whatever you can dream up. All you have to do is read and write files to a local directory, Cup (and Wazero) will take care of the rest.

If you’re interested in seeing this in action, we developed a controller you can explore for [Flipt feature flag configuration](https://github.com/flipt-io/cup/tree/main/ext/controllers/flipt.io). This is a stepping stone we envision for creating a UI experience for Flipt over Git. The video on the [Cup GitHub project README](https://github.com/flipt-io/cup#-feedback) demonstrates the Flipt controller for Cup.

### Future Vision

We’ve thrown around a lot of potential projects that could be built on top of Cup and Git in the future. Including but not limited to:

- Internal developer platforms
- Continuous delivery automation and observability
- Automatic dependency updates
- Progressive delivery
- End consumer self-service of infrastructure (let your customer's actions invoke configuration change)
- General purpose configuration API for your applications (why not read from Cup directly in your applications)

To get there we’re going to need to build some more features. Our current roadmap of ideas includes:

- Proposal status directly in the Cup API (see what desired state has been requested)
- Authentication and authorization (fortunately we have a consistent API to hang policies on)
- First-class clients in different programming languages
- Automatic replay of desired state changes as track branches progress (allowing for proposal to stay accurate as Git commits are merged ahead of your desired changes)
- Better management and packaging for controller implementations (support pulling controllers directly from OCI registries)

There is a lot to do to prove (or disprove) the idea, but the project is out there and we’re committed to exploring it. We could really do with your expertise in evolving this idea. Is there something you could automate with the help of Cup? We want to hear about it and to make Cup more appropriate for you.

Come check out the project at [https://github.com/flipt-io/cup](https://github.com/flipt-io/cup) and open an issue or discussion. Alternatively, come speak to us in realtime on our [Discord](https://flipt.io/discord).


GitOps Is a Means, Where Is the End?


Feature flags have become ubiquitous in the modern software development lifecycle for organizations across the world. Decoupling features from deployment, A/B testing, and gradual rollouts of features are a few of the benefits that come along with using feature flags.

Check out our [usescases](https://www.flipt.io/docs/usecases) documentation for more information on how feature flags can be used in your organization.

Feature flags allow for safety and ease of mind when rolling out features, but what happens when this new functionality behind feature flags are ready to be deployed fully? What tools do you need in place to monitor for unexpected application behavior? How can the organization correlate flag changes with regressions, or erratic results in their application behavior?

These questions demonstrate how there is a need for tighter coupling of feature flags to obvservability tooling. As a feature flaging tool at our core, we believe that we can introduce functionality to make this easier on our users and integrators to achieve tighter correlation with what can be observed while running in production. This blog post will highlight some of the features we have recently implemented to streamline that correlation.

### Webhooks

In the [v1.21.0](https://github.com/flipt-io/flipt/releases/tag/v1.21.0) release, Flipt introduced the [Audit Sink](https://www.flipt.io/docs/configuration/observability#audit-events) functionality where users can configure events in Flipt to be sent to a sink for auditability.

That release came with implementation for the inaugural sink which was a log file. These events are all CUD (create, update, delete) operations on Flipt entities (`flags`, `variants`, `segments`, `constraints`, `rules`, `rollouts`, `distributions`, `tokens`).

The log file was easy enough to get started, but hard to centralize that data to make any use out of it. The user would have to have additional tooling, such as [Grafana Loki](https://github.com/flipt-io/flipt/tree/50edfb1b78e02081b8c6038129f2f7903d39646c/examples/audit), to gather the logs in a centralized location since the logs can be distributed amongst many Flipt instances.

<BlogImage
  src="/observability/loki.png"
  alt="Audit Events Aggregated in Grafana Loki"
  width={1776}
  height={1304}
  className="shadow-md"
  caption="Audit Events Aggregated in Grafana Loki"
/>

As of [v1.27.0](https://github.com/flipt-io/flipt/releases/tag/v1.27.0), we have released functionality for a sending audit event payloads to a webhook sink. 🎉

Webhooks are a common way for applications to send data to a URL when an event happens. If you are familiar with GitHub, you may have seen the [webhooks](https://docs.github.com/en/developers/webhooks-and-events/about-webhooks) functionality that allows you to configure a URL to send events to when certain actions happen on a repository. This is the same concept in Flipt.

<StarCTA />

You can configure a URL with the [`audit.sinks.webhook.url`](https://www.flipt.io/docs/configuration/overview#audit-events-webhook) value and start receiving events as they happen to the web server that the URL is pointing to.

Use cases include: get notified in a Slack channel when a new flag is created, or transform and post that event to an [Events API](https://docs.datadoghq.com/api/latest/events/#post-an-event) in your observability solution (ie: Datadog). The webhook sink also can be configured with a signing secret to allow users to validate the event originated from somewhere appropriate.

<BlogImage
  src="/observability/webhook.png"
  alt="An Audit Event sent to a Webhook Sink"
  width={2880}
  height={1800}
  className="shadow-md"
  caption="An Audit Event sent to a RequestBin Webhook Sink"
/>

You can start using this functionality today! Check out the [documentation](https://www.flipt.io/docs/configuration/observability#webhooks) for more information as well as [an example](https://github.com/flipt-io/flipt/tree/main/examples/audit/webhook) in our main repository.

### Datadog

Datadog has long established themselves as a giant in the observability SaaS space. Knowing this, we figured that our users would benefit from tighter integrations with Datadog. The following sections will highlight some of the features we have implemented to allow for an improved observability experience if using Flipt with Datadog.

#### Exporting OTLP Traces

Flipt already has the ability to emit [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/) traces to a configured collector. This depends on you spinning up the OTLP collector in your stack, and using the URL for that collector as configuration for Flipt. Until now however, we havent done a great job documenting how you could send your OTLP traces to third party tool such as Datadog.

We have recently published [an example](https://github.com/flipt-io/flipt/blob/cac1e65026add36cce51196fa9007eb4d235a5ae/examples/tracing/otlp/README.md#datadog-ui) on how to configure tracing with Datadog through the OTLP collector. Once the traces arrive in Datadog, you can leverage built-in features to [correlate traces and logs](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/), and [logs with metrics](https://docs.datadoghq.com/logs/guide/correlate-logs-with-metrics/), a nice two-way correlation. As things regress or errant things happen in your application, you can use these features to quickly track down a problem, and possibly correlate them to a switch in a feature flag.

<BlogImage
  src="/observability/datadog.png"
  alt="Flipt Traces in Datadog"
  width={1691}
  height={746}
  className="shadow-md"
  caption="Flipt Traces in Datadog"
/>

#### RUM

Many times organizations want a better view of not only what occurs server-side in their applications, but also on the front-end. For example, you might want to see exactly what steps a user took during a session, where the user might have navigated away (bounced), and a variety of other metrics. Organizations can use feature flags to supplement this especially when they are rolling out new UI functionality to see how a subset of users might react without rolling it out completely to their entire userbase.

[Datadog Real User Monitoring (RUM)](https://www.datadoghq.com/dg/real-user-monitoring/benefits-na/) is a drop in client that you can use within your front-end to capture the metrics that you care about during a user's session. The RUM client also has functionality for tracking the values of feature flags and correlating that with the metrics that the RUM client already keeps track of.

As of [v0.2.11](https://github.com/flipt-io/flipt-node/releases/tag/0.2.11) of our [Node/TypeScript client](https://github.com/flipt-io/flipt-node), we now have the ability to integrate with Datadog RUM natively.

Here's an example of how you can use the client to send evaluated values of feature flags to Datadog RUM:

```typescript
const metricsClient = new FliptMetrics(
  new FliptApiClient({
    environment: "http://localhost:8080",
    auth: {
      credentials: {
        username: "YOUR_USERNAME",
        password: "YOUR_PASSWORD",
      },
    },
  }).evaluation,
  datadogRum
);

const response = await metricsClient.variant({
  namespaceKey: "default",
  flagKey: "hello-this",
  entityId: uuidv4(),
  context: {},
});
```

The [README](https://github.com/flipt-io/flipt-node#metrics) in the repository goes a bit more in depth of how you can enable that functionality.

<BlogImage
  src="/observability/datadog-rum.png"
  alt="Datadog RUM with Flipt"
  width={2704}
  height={1560}
  className="shadow-md"
  caption="Datadog RUM with Flipt"
/>

The above image shows an instance where the flag `hello-this` in the `default` namespace was `enabled` during the entirety of a user's session. On the same Datadog UI page, you can click on the `Session & Replays` tab to see how that new functionality behind the feature flag may have impacted the user session compared to users who the feature flag was off or evaluated `false` for.

## Conclusion

With these new observability focused features, we hope to allow our users to gain more visibility into how feature flags are affecting their application/platform as a whole.

If you have any questions, other tools you'd like us to integrate with, or additional features you'd like to see added, please reach out to us on [GitHub](https://github.com/flipt-io), [X](https://twitter.com/flipt_io), or [Discord](https://flipt.io/discord).

We hope you enjoy these new features, and we look forward to hearing from you!


Improving Observability


Last week at Flipt, we took a break from our regular projects to kick off our first company-wide Hack Week. If you're unfamiliar with the concept of Hack Weeks, they are dedicated periods during which teams can work on creative and innovative projects outside their regular responsibilities. It was an excellent opportunity for everyone to stretch their creative muscles and explore new possibilities.

This post will cover the projects we developed, the insights we gained, and how these innovations might fit into our broader company vision.

We had three days to work on our hack projects, starting on Tuesday and culminating in a demo day on Friday. I'm incredibly proud of what the team accomplished in such a short period of time. Without further ado, let's dive into the projects.

## Project 1: Flipt Migrator

[Mark Phelps](https://www.linkedin.com/in/markphelps1/)

<BlogImage
  src="/hackweek/migrate.gif"
  alt="Flipt Migrator"
  width={1070}
  height={934}
  className="shadow-md"
/>

I wanted to simplify the process for users migrating from existing solutions to Flipt. One of the biggest challenges teams face after deciding to switch tools is the absence of a straightforward migration path. Teams often either:

1. Run the new solution in parallel with the existing one, leading to inconsistent data between the two systems.
2. Spend considerable time and effort writing custom migration code, sacrificing productivity that could be better spent on core business activities.

To address these issues, I created [Flipt Migrator](https://github.com/flipt-io/migrator), a CLI tool that automates the migration process. The tool extracts data from the existing solution, transforms it into a Flipt-compatible format, and allows teams to validate and import the data.

Above is a short .gif demonstrating the migrator, where I migrate data from a competitor into a format compatible with Flipt and validate it using our new `flipt validate` command.

The tool, written in Python, currently supports only one source but we open-sourced it to encourage community contributions. Check it out on [GitHub](https://github.com/flipt-io/migrator)!

It leverages the [Pydantic library](https://docs.pydantic.dev/latest/) for type validation in Python3, making data transformation more reliable and intuitive.

Our goal is to make Flipt the best solution for feature management, and that includes making it easy to migrate from other solutions. We're excited to see how this tool evolves over time.

## Project 2: Slack Bot

[Yoofi Quansah](https://www.linkedin.com/in/redblacktree/)

<BlogImage
  src="/hackweek/slack-1.gif"
  alt="Flipt Slack Bot Create Feature Flag"
  width={1510}
  height={844}
  className="shadow-md"
/>

Yoofi's project was a Slack bot designed to centralize communication and auditing for feature flag changes. In organizations where multiple teams communicate via Slack, feature flags often require cross-team coordination. Existing solutions lacked a centralized system for auditing and tracking these changes. Our Slack bot fills this gap by integrating feature flag management directly into Slack.

The bot operates through a custom server-side application that interfaces with Slack. Users configure Slack to send specific event notifications to this application. The application then performs various tasks, such as serving block kit forms back to Slack for creating new feature flags or listing existing flags within a specific namespace.

The Slack bot has the potential to become an indispensable tool for broader platform management within Flipt. We're exploring additional API functionalities and the possibility of enhanced authorization through OAuth or OIDC providers.

Outside of Slack, we've also got some new auditing features in the works that go beyond our existing [file-based event sink](https://www.flipt.io/docs/configuration/observability#audit-events), so stay tuned for more updates!

## Project 3: Ready - Feature Flight Plans

[George MacRorie](https://www.linkedin.com/in/geokeaton/)

<BlogImage
  src="/hackweek/ready.gif"
  alt="Ready Feature Flight Plans"
  width={1068}
  height={720}
  className="shadow-md"
/>

Developed by George MacRorie, our third project is "Ready," an app designed to standardize the way engineers plan and execute feature releases.

The journey from code completion to general availability is complex and varies from one organization to another. It often relies on word-of-mouth knowledge or outdated internal documentation. Ready aims to define this process, integrate all relevant tools, and build an executable flight plan for releases.

Ready is a task list focused on feature releases, offering rich integrations with essential production tools. It guides developers through a tailored process, allowing plans to be created, templated, and shared within the organization. A central dashboard provides visibility into active features and their current state.

Over time, Ready could become the go-to tool for getting new features into production. Some ideas include integrating more source control management systems, observability tools, and deployment solutions. Ultimately Ready could evolve to include autopilot features to automate actions at each step, allowing for a more streamlined and less error-prone process.

George also prepared a longer Loom video demonstrating the app. Check it out [here](https://www.loom.com/share/3d86a9d9864a43228e21a99ed5d5a0b3) to see Ready in action!

We're super exicted about the potential of this project and how it could help teams streamline their release processes beyond feature flags.

## Until Next Time

This first Hack Week was a great success, and we're already looking forward to the next one.

We hope you enjoyed this brief overview of our projects. If you have any questions, feedback, or want to try any of them out for yourself, please reach out to us on [Twitter (X)](https://twitter.com/flipt_io) or [Discord](https://flipt.io/discord). We'd love to hear from you!

Also, signup for our monthly [newsletter](https://www.flipt.io/#newsletter) to stay up to date with all things Flipt including new features, blog posts, and more!


Flipt's Inaugural Hack Week


Today we’re happy to announce the release of [v1.24.0](https://github.com/flipt-io/flipt/releases/tag/v1.24.0) of Flipt including a few major features including:

- A new `boolean` flag type that can be used in combination with [flag rollouts](https://www.flipt.io/docs/concepts#rollouts) allowing you to enable a flag over a percentage of your user base and more
- A new, more straightforward set of evaluation API routes that you can also [enable/disable authentication](https://www.flipt.io/docs/configuration/authentication#exclusions) separate from the existing ‘management’ APIs
- The ability to add user email addresses to [an allowlist](https://www.flipt.io/docs/configuration/authentication#email-matches) when setting up OIDC authentication
- Improved performance and efficiency in flag evaluation with cache enabled
- New [Amazon S3 filesystem backend support](https://www.flipt.io/docs/experimental/filesystem-backends#object) to serve your flags directly from an S3 bucket

## Boolean Flag Type and Rollouts

Previously, Flipt was designed with only a single flag ‘type’ in mind, a flag which can be enabled/disabled and optionally have variants that can be returned during evaluation time. While this worked well for many use cases, we recognized the need for more flexibility. With the introduction of the new `boolean` flag type, you can now enable a flag over a percentage of your user base more easily. This allows for better experimentation and targeted rollouts.

The `variant` flag type is still the default flag type, allowing you to use flags in the same way as before. The new `boolean` flag type adds an extra layer of versatility. Boolean flags work hand in hand with a new abstraction that we call 'rollouts' or 'rollout rules'.

[Rollouts](https://www.flipt.io/docs/concepts#rollouts) work similarly to rules used during evaluation of a `variant` flag in that the first rule that matches ‘wins’, however there are a few key differences:

1. Rollouts are only allowed for `boolean` flag types and allow you to return a different value for a flag’s `enabled` property.
2. There are currently two types of rollouts: **threshold** and **segment**. Segment rollouts work just like previous rules in that you specify a segment that you want the rule to match. Thresholds, in contrast, are used to return a different result to a percentage of requests.
3. Rollouts allow you to specify the value to return for a boolean flag, either `true` or `false`.

It’s perhaps easier to show an example of the problem that boolean flags and rollouts were added to solve, by showing how you would have had to configure Flipt pre-v1.24.0 without rollout support and then how you would do the same now.

**Imagine the following scenario:**

You have a feature in your application that you want to roll out in stages to all users. Before the release of v1.24.0 of Flipt, you would have had to:

1. Create a flag with `true` and `false` variants, or `on`/ `off` or similar
2. Manually handle these variants in your application code, converting them to `booleans`, since `variants` can only be strings.
3. Configure a segment with constraints that matches the segment you want to target (ie: `all-users`)
4. Create an evaluation rule which returns the `true` variant to X% of requests, and the `false` to the remaining percentage.

<BlogImageGallery
  images={[
    {
      src: `/rollouts/rollout-variants-1.png`,
      alt: "Creating the Flag and Variants",
    },
    {
      src: `/rollouts/rollout-variants-2.png`,
      alt: "Creating the Segment to Match",
    },
    {
      src: `/rollouts/rollout-variants-3.png`,
      alt: "Setting up the Evaluation Rule",
    },
    {
      src: `/rollouts/rollout-variants-4.png`,
      alt: "The Flag with Variants and Evaluation Rule",
    },
  ]}
/>

As you can see , managing rollouts with boolean flags in Flipt used to be quite complex. However, with the release of [v1.24.0](https://github.com/flipt-io/flipt/releases/tag/v1.24.0), the process has become much simpler and more intuitive.

Now, to achieve the same rollout scenario, you can follow these steps:

1. Create a `boolean` type flag. This flag will represent the feature you want to roll out.
2. Configure a threshold-based rollout, specifying the percentage of requests that should receive the flag enabled. The remaining percentage will receive the flag disabled.
3. Thats it! 🎉 You can now use the flag in your application code as a boolean value.

<BlogImageGallery
  images={[
    {
      src: `/rollouts/rollout-boolean-1.png`,
      alt: "Creating the Boolean Flag and Rollout",
    },
    {
      src: `/rollouts/rollout-boolean-2.png`,
      alt: "The Boolean Flag and Rollout Rule",
    },
  ]}
/>

Not only are boolean flags easier to setup for this scenario, they are also easier to update in place in the UI. You can now simply drag the slider or manually input the percentage in the rollout input box and click update to increase or decrease the percentage over time!

<StarCTA />

## Evaluation API Routes and Security Configuration

While designing and implementing boolean flags, we also took the time to see how we could make evaluation more intuitive to support both flag types. We decided to create a new set of evaluation API routes, separate from the existing `/api/v1` prefix that can work with both variant and boolean flags.

Several users have previously asked how to separate the ‘management’ API (created/edit/delete flags and other resources) from the ‘evaluation’ API as they serve two distinct audiences. Normally the ‘management’ API is used by team members to setup Flipt resources and should not be exposed publicly. Depending on a team’s application architecture however, they may need to evaluate flags from ‘untrusted’ and unauthenticated requests, such as when using feature flags directly from a browser based application.

Before this release, our only answer was to suggest putting Flipt behind a reverse proxy such as Caddy or Nginx and configuring access rules there. While we still think reverse proxies are a good idea for handling things such as request routing, rate limiting, and more, v1.24.0 of Flipt now separates the ‘management’ and ‘evaluation’ APIs completely and allows the operator to enable or disable authentication for each independently. We think that this new set of evaluation APIs and authentication configuration will allow Flipt to work for a larger variety of applications and use cases than before.

Checkout the new [evaluation API documentation](https://www.flipt.io/docs/reference/overview#v1-24-0) as well as all of our updated clients which now also support these new endpoints!

We also added the ability to configure `allowed_emails` when setting up OIDC Authentication for Flipt. This addition was driven by a [user request](https://github.com/flipt-io/flipt/issues/1872) to be able to configure specific users within their organization to have access to Flipt when configuring authentication with Google.

**Note:** in order for this feature to work properly, make sure you specify the `email` scope when [configuring your OIDC setup](https://www.flipt.io/docs/configuration/authentication#email-matches) for Flipt.

## Improved Performance, S3 Backend and more

Finally, we've merged two amazing user-contributed PRs which made significant improvements to flag evaluation performance and efficiency as well as added support for serving flag state straight from an S3 bucket!

### Faster Evaluation

With cache enabled, the evaluation of variant flags has become even quicker as we now cache potential constraints that would match a request at the storage layer, meaning they can be used across different `entityIds`.

From the [PR](https://github.com/flipt-io/flipt/pull/1910):

> after some investigation we found the main cause of delays in evaluation requests were the rules query, in our case running Flipt with a PostgreSQL DB on AWS RDS we were getting up to 1s of delays, with this fix it reduced to a few milliseconds

We consistently aim to keep Flipt fast for our users, and thus your users so we very much appreciate any contributions that help improve performance!

### Flag State Not Just for Git Anymore

Last but certainly not least, Flipt now supports the use of [Amazon S3 as a filesystem backend](https://www.flipt.io/docs/experimental/filesystem-backends#object) to serve your flags directly from an S3 bucket or set of buckets.

In [v1.23.0](https://github.com/flipt-io/flipt/releases/tag/v1.23.0) of Flipt we released a new set of experimental features allowing our users to manage their Flipt state. We strongly believe that feature flags should be treated like any other application configuration in our your infrastructure, which means you should also be able to use industry best practices such as managing configuration via CI/CD pipelines with GitOps. We also recognize however that every organization is different and so we’re delighted to now support Amazon’s S3 as an alternative filesystem backend!

This was another amazing user contribution. Here’s the reasoning for the feature straight from the [PR](https://github.com/flipt-io/flipt/pull/1900):

> In an AWS environment, this would allow deploying a readonly container and not require communication to a git repository outside of the deployment environment.

This feature opens up the door for us to implement other object stores such as Azure Blob Store and Google Cloud Storage as well, so please [open an issue](https://github.com/flipt-io/flipt/issues/new/choose) if you’d like us to support those backends as well!

## That’s a Wrap

This release is our most feature packed yet! We’re delighted to add functionality that makes Flipt better for all of our users, and love the community contributions we’ve gotten so far this summer.

As always, feel free to reach out on [Twitter](https://twitter.com/flipt_io) (X?), [Discord](https://flipt.io/discord), or open a [GitHub issue](https://github.com/flipt-io/flipt/issues/new) for any functionality that you’d like to see implemented. We’d also love to answer any questions or just chat about software delivery in general.

Checkout the full [changelog](https://github.com/flipt-io/flipt/releases/tag/v1.24.0), and give the latest release a look and let us know what you think!


Boolean Flags and Rollout Functionality


<BlogImage
  src="/labs/flipt-labs.png"
  alt="Announcing Flipt Labs"
  width={2400}
  height={1350}
>
  <span>
    Image generated by <a href="https://www.midjourney.com/">MidJourney</a>
  </span>
</BlogImage>

Today we’re excited to show off a new project we’ve been working on. We’re calling it [Flipt Labs](https://github.com/flipt-io/labs). This project aims to show off everything you can do with Flipt by providing examples of real-life applications and how they can integrate with Flipt.

## Applications as Tutorials

Creating high-quality tutorials takes effort and dedication. We believe that good tutorials can make a big difference in helping users fully leverage the power of Flipt. That's why we decided to create Flipt Labs, where we can showcase different applications and really put ourselves in our user’s shoes.

One of the challenges in demonstrating an application such as Flipt is that it's not entirely useful on its own. To see the real benefit of using feature flags, it's necessary to integrate them within applications to demonstrate practical use cases.

Our first full application in the Labs repository is an interactive tutorial that steps the user through several ways of using Flipt:

1. Using flags for basic on/off global release
2. Using segmentation and evaluation to serve different responses based on request context
3. Using our new [GitOps backend](/gitops-announcement) to support managing your feature flags in your Git repository

We decided that the tutorial would be based around the scenario of an engineer wanting to experiment with and eventually release a chatbot to their end users. We got a bit cheeky and chose to sprinkle in some LLM/AI magic as well to keep things interesting. ✨

## Try it Out

<BlogImage
  src="/labs/chatbot.png"
  alt="Flipt Chatbot Application"
  width={1692}
  height={1124}
></BlogImage>

Check out our [first full demo application](https://github.com/flipt-io/labs/blob/main/chatbot/README.md) and tutorial on GitHub. You can run it locally via NPM + Docker Compose, or in a GitHub Codespace. We would love to hear your feedback, so please let us know what you think!

We're working on another post that will go into more detail about how we built a working chatbot with context about Flipt itself! In the mean time, check out the code to see examples of using both the [Flipt Node SDK](https://github.com/flipt-io/flipt-node) for the frontend and the [Flipt Python SDK](https://github.com/flipt-io/flipt-python) for the backend, as well as how our [GitOps Backend](https://www.flipt.io/docs/experimental/filesystem-backends#git) can integrate into your existing workflow.

## Onward

We believe that this project will highlight the potential of Flipt and provide inspiration for its integration in various contexts. By showcasing practical examples, Flipt Labs aims to empower users to unlock the full potential of Flipt in their own projects. Through this initiative, we hope to encourage creativity and innovation in utilizing Flipt's features and capabilities.

If you have any ideas for applications you would like to see, please let us know! Feel free to open an issue on the [Flipt Labs](https://github.com/flipt-io/labs) repository or reach out to us on [Discord](https://flipt.io/discord) or [Twitter](https://twitter.com/flipt_io).

Stay tuned for updates as we release new projects and tutorials! Happy experimenting 🧪.


Announcing Flipt Labs


<BlogImage
  src="/gitops/flipt-gitops.png"
  alt="Flipt for GitOps"
  width={2400}
  height={1350}
>
  <span>
    Image generated by <a href="https://www.midjourney.com/">MidJourney</a>
  </span>
</BlogImage>

Today we’re excited to announce that we’re exploring a new type of backend for Flipt!

The team has been discussing and exploring ways Flipt could be better positioned to support GitOps-style workflows. For those unfamiliar with GitOps practices, we recommend the WeaveWorks [Guide To GitOps](https://www.weave.works/technologies/gitops/).

Our new [filesystem backends proposal](https://github.com/flipt-io/flipt/discussions/1652) adds two ways to configure Flipt to serve flag state content encoded as yaml stored in files, `local` and `git`.

We’re keen to understand if this idea resonates with you and your use case, this will help us shape the implementation, which is currently underway.

## Problem

We feel that many feature flags offerings (including Flipt today) work at odds with GitOps practices. GitOps prescribes that you store all your configuration in one or more Git repositories. This gives your entire system the benefits of version control. Feature flag configuration is just like any other configuration in that it adjusts the behaviour of your applications at runtime. However, we find that it is rarely stored and tracked within Git itself.

If you’re using GitOps practices for your wider configuration needs, but your feature flag solution is completely isolated, then you’re subverting the benefits and traceability of Git. When the alerts start firing you'll have to correlate your configuration changes in Git with the auditing features of your flagging solution.

Alternatively, some existing feature flag solutions support GitOps by bolting a declarative syntax onto their existing imperative API. For example, some offerings provide a [Terraform](https://www.terraform.io/) provider which is an entirely legitimate way to create a GitOps-style contribution experience. In fact, we’re still considering having one for Flipt and our existing relational backends. However, this approach does have drawbacks, primarily, there are now two sources of truth.

Reconciling from your Git repository into your online feature flag service also has to be performed. This process is often performed during some kind of continuous delivery step (e.g., via a GitHub action), which likely pushes changes into your feature flag system. You may have to manually intervene when this process fails for intermittent reasons. Your environment must also have credentials and access to your application repository and your feature flag solution. Ideally, your feature flag solution provides tooling to ease the extra work required to set up this CI/CD process and make it resilient.

In this scenario, given you’re using a self-hosted solution (such as Flipt), you now have the following set of applications to run and configurations to manage:

- The feature flag service itself (e.g. the `flipt` binary)
- Backing database for storing flag state (e.g. SQLite, PostgreSQL etc)
- Flag state configuration (e.g. in Terraform HCL or via some other process such as `flipt import`)
- CI processes for reconciliation (plus monitoring)
- For solutions such as Terraform, storage of reconciliation state (`tfstate` files)

This is all achievable, but care must be taken when building the reconciliation process to ensure a predictable end state.

Ultimately, feature flag offerings have to build a host of features you’re already getting from your SCM (e.g. GitHub or Gitlab). Version control, peer review, access control and backup to name a few.

## How We Intend to Serve GitOps

We intend to have Flipt pull feature flag state directly from Git at runtime.

We want Flipt to be able to drop its dependency on a database for flag state and go directly to the source. This removes the need for a reconciliation process altogether. By adopting a pull model, we embrace eventual consistency and allow Flipt to recover automatically during transitive errors without intervention.

You'll simply be able to configure Flipt with access to your repository and with some additional configuration, you will be able to control how and where to locate your flag state files. Your repository could be used solely for Flipt, it could be your existing configuration repository, or it could be stored within a monorepo.

<BlogImage
  src="/gitops/gitops-scm.png"
  alt="Flipt for GitOps - SCM"
  width={800}
  height={600}
/>

Flipt’s evaluation and read APIs won't change, meaning your existing interactions with the Flipt SDKs can remain the same. The existing [`flipt export`](https://www.flipt.io/docs/configuration/storage#export) command produces configuration files valid for committing straight to your repositories for ease of transition.

Flipt will be configured to serve a particular branch for a target repository. This branch doesn’t always have to be `main`, meaning you can deploy Flipt in multiple ways to support validating flag state changes in arbitrary Git branches or tags.

You'll also be able to run Flipt over a local directory. In this configuration, Flipt will simply serve your flag configuration without the need to go through Git. Updates to the local directory will update Flipt automatically. There is no need to access an external running instance of Flipt to replicate or alter your production feature flag state in your _local_ environment. Simply check out your repository and run `flipt` to begin experimenting.

<BlogImage
  src="/gitops/gitops-local.png"
  alt="Flipt for GitOps - Local"
  width={800}
  height={600}
/>

## Flipt in CI

Alongside building a new core backend for Flipt, we’re also developing a suite of tools for validating your use of Flipt during your existing continuous integration workflows.

Before Flipt can serve your configuration files, it is important to validate they are fit for consumption. Failure to do so could lead to runtime errors or unexpected behaviour. `flipt validate` is a new subcommand currently being added to Flipt. This will be useful for local validation or running directly in a CI pipeline.

We’re also developing a GitHub action which simplifies running `flipt validate` which will make it trivial to add alongside your existing GitHub workflows.

## Just the Beginning

We’re just getting started turning our proof-of-concept into something ready for production Flipt usage. While we have a good feeling that the experience will resonate with users, there are still a number of interesting technical challenges we expect to have along the way. This is why we wanted to get this plan out early in order to hear your feedback.

Help come shape the future of this solution on [GitHub](https://github.com/flipt-io/flipt/discussions/1652), or in our [Discord](https://flipt.io/discord).


How's about Git?


<BlogImage
  src="/audit-events/audit-events.png"
  alt="Events for Flipt"
  width={2400}
  height={1350}
>
  <span>
    Image generated by <a href="https://www.midjourney.com/">MidJourney</a>
  </span>
</BlogImage>

Events and logging are vital for many organizations to either keep track of history within the application/platform or to actually take action on things once they happen. Two brief examples include:

1. An organization keeping track of payment events that have happened on their platform
2. A platform triggering a job to purge data somewhere if an event is of a particular nature

In Flipt's case it may prove essential for some organizations to know when feature flags are mutated in some fashion as they can affect the real-time behavior of the consumer’s application.

It's also important to know who made the change and when it was made, especially when it comes to trying to determine the root cause of an issue in the consuming application. This is where audit events come in.

We're happy to announce that support for these type of audit events are now implemented as of [v1.21](https://github.com/flipt-io/flipt/releases/tag/v1.21.0) of Flipt! 🎉

## Ideal Implementation

Before diving into how the feature was implemented, it is worth discussing what an ideal scenario looks like for Flipt and audit events. Many providers, whether SaaS or open source, leave it strictly up to the consumer to deal with events as they please. In most cases, this is done through a webhook URL that you provide to the platform which sends a request to that URL when an event occurs.

An application that is listening to the endpoint of the webhook URL can then do whatever they want with the event. For instance, they can choose to send the message to a Kafka broker, do some other sort of pub/sub, or send that audit event to Slack in a specialized channel, so this is more of the laissez-faire approach.

What we decided on for an ideal implementation is to have this concept of first-class support for ‘native sinks’. In this case, Flipt will house the logic necessary to interact directly with the sinks that the user provides configuration for. The idea is that messages can go straight to a Kafka broker, pub/sub system, Slack channel, etc, as a direct form of communication. This allows users to avoid having to write their own backend to communicate with these systems and avoid the extra network hop.

The implementations for communicating with these native sinks would ideally have an abstraction point so that users can make easy sense of it to contribute to Flipt and write their own implementations for native sinks. One of the downsides of this native approach, however, is that as more and more sinks become implemented, it can lead to version churn for the Flipt application. To combat this potential for churn and to make contributions more accessible, we are currently looking into a way to make the sink implementations pluggable in their own repositories, potentially using a plugin system like Hashicorp's [go-plugin](https://github.com/hashicorp/go-plugin).

## Implementation

Implementing any event-driven system comes with many considerations and trade-offs. On the producer side of the system, an ‘event’ usually represents things in the system that have already happened to then notify the consumer. The benefit for the consumer(s) is that once the event is received on its end it knows the event is complete, and therefore does not have to poll and waste resources getting the complete event from the source.

While the definition of an event-driven system seems simple enough, there are many considerations to keep in mind. In this case particularly, what happens to the producer or consumer during high-traffic scenarios? Or what if the consumer becomes unavailable for a portion of the time? These represent just a few of many considerations, and we sought to address those in our implementation.

<BlogImage
  src="/audit-events/event-diagram.png"
  alt="OTEL Event Diagram"
  width={300}
  height={261}
  caption="The two basic components of an event system. One has to consider where to implement the functionality to deal with the considerations described above."
/>

Our first stab at an implementation was a basic homegrown publisher which contains abilities to batch events and send batches asynchronously to different sinks that are configured. This is done so the consumer is not burdened with a whole bunch of writes, and it frees up the publisher to do other work without worry of communication failures with the consumer (addressing some of those considerations mentioned above).

In addition to a batch size, there should also be configuration for a flush period to avoid messages laying around in memory for an indefinite amount of time. Considering all of this, it was a pretty straightforward implementation, but halfway through we realized that this problem has been solved several times for a lot of different use cases, leading us to pause the publisher implementation and seek out a more tried and true solution.

### Enter OpenTelemetry

<BlogImage
  src="/audit-events/otel.png"
  alt="OpenTelemetry"
  width={1000}
  height={524}
/>

For the sake of brevity, [OpenTelemetry](https://opentelemetry.io/) will not be discussed in this article, but simply how it was used in the Flipt application to achieve desired functionality for audit events. For trace data, OpenTelemetry (OTEL) not only provides configurable batching, and flushing abilities but also allows users to provide implementations of [`Exporters`](https://opentelemetry.io/docs/concepts/glossary/#exporter) which OTEL uses to send trace data to whatever the implementation specifies. In simpler words... OTEL does everything already implemented in the homegrown publisher implementation, and it already exists as a dependency in the Flipt source code for exporting trace data to [Jaeger](https://www.jaegertracing.io/), [Zipkin](https://zipkin.io/), or [OTLP](https://opentelemetry.io/docs/reference/specification/protocol/).

There might be some lack of context as to what a `Trace` or `Span` here is and how it ties into audit events for Flipt. Essentially, a `Trace` represents a unique transaction of a collection of operations throughout an application. A `Span` represents a single unit within that `Trace` or a single operation. Spans allow for additions of [Events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events) which are viewable through any tracing provider client. The addition of `Event`s is the bit that we leveraged for audit events.

Specifically in the code, we’ve implemented a middleware that will intercept the request, and determine if the event is auditable, if it is, the code will add the details of that audit event to the Span via the `Event` described above. When a batch size or flush period is reached for the `SpanProcessor`, events are pulled off of the `Span`(s) and converted into a data structure housed in our code to then send it off to the native sinks that the user has configured on their Flipt instance. In addition to viewing the audit events in their sink implementation, an added benefit of using OTEL is the span events will show up in the tracing provider client for users to view.

<BlogImage
  src="/audit-events/jaeger-ui.png"
  alt="Jaeger UI with Event"
  width={3024}
  height={1542}
  caption="The audit event added to the flipt.Flipt/CreateConstraint span. The flipt* properties represent the audit event."
  className="shadow-md"
/>

## What's Released

With this initial release, there is only functionality for audit event logging to a file on disk. Users can specify the name of that log file, and Flipt will write the audit events to it in a JSON-encoded format.

If you were to `tail` the logs of your audit event log file for orthodox Flipt usage you might see something like the following:

```json
{"version":"0.1","type":"flag","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"description":"your favorite NBA team","enabled":true,"key":"scenter","name":"scenter","namespace_key":"default"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"variant","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"attachment":"","description":"","flag_key":"scenter","id":"41e84846-bfbd-426f-afa9-3358d0d40875","key":"lakers","name":"","namespace_key":"default"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"segment","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"constraints":[],"description":"Segments for evaluation NBA teams","key":"postclavicula","match_type":"ANY_MATCH_TYPE","name":"postclavicula","namespace_key":"default"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"constraint","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"id":"a7ccde21-870e-4e36-953c-4d5849d96a77","namespace_key":"default","operator":"eq","property":"championship","segment_key":"postclavicula","type":"STRING_COMPARISON_TYPE","value":"most"},"timestamp":"2023-05-01T14:01:12-05:00"}
{"version":"0.1","type":"flag","action":"updated","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"description":"your favorite NBA team","enabled":false,"key":"eyepoint","name":"eyepoint","namespace_key":"default"},"timestamp":"2023-05-01T14:01:37-05:00"}
{"version":"0.1","type":"token","action":"created","metadata":{"actor":{"authentication":"none","ip":"127.0.0.1"}},"payload":{"io.flipt.auth.token.description":"hello","io.flipt.auth.token.name":"worldh"},"timestamp":"2023-05-01T14:02:45-05:00"}
```

Currently, the auditable events are all CRUD operations (sans Read) upon the following entities:

- [Flag](https://www.flipt.io/docs/concepts#flags)
- [Variant](https://www.flipt.io/docs/concepts#variants)
- [Distribution](https://www.flipt.io/docs/concepts#distributions)
- [Segment](https://www.flipt.io/docs/concepts#segments)
- [Constraint](https://www.flipt.io/docs/concepts#constraints)
- [Rule](https://www.flipt.io/docs/concepts#rules)
- [Token](https://www.flipt.io/docs/authentication/methods#static-token)
- [Namespace](https://www.flipt.io/docs/concepts#namespaces)

The event payload which is logged has the following fields for each event:

- `type`: the type of entity changing
- `action`: the action taken upon the entity
- `metadata`: extra information about the event such as identity of the subject who initiated the event
- `payload`: the actual payload of the request on Flipt
- `timestamp`: the timestamp of when the event was created

Any combination of actions on those entities will result in an event with the above fields being logged out to the file you specify via configuration:

```yaml
audit:
  sinks:
    log:
      enabled: true
      file: /tmp/flipt/audit.log
```

We also have an example of how you could potentially collect, parse, visualize, and query these events using Grafana's [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) + [Loki](https://grafana.com/docs/loki/latest/) in our [examples](https://github.com/flipt-io/flipt/tree/main/examples/audit) on GitHub. This would allow you to query the audit events in a similar fashion to how you would query logs in a traditional logging system, even when Flipt is running in a distributed fashion.

<BlogImage
  src="/audit-events/loki.png"
  alt="Grafana Loki Query"
  width={2880}
  height={1800}
  caption="Querying Flipt audit events using Grafana Loki"
  className="shadow-md"
/>

## What's Next

We are excited for this release to see just how users will interact with this feature, and also excited for community discussion on potential event sinks that have not been added yet.

[Contributions](https://github.com/flipt-io/flipt#contributing) are always appreciated! This [document](https://github.com/flipt-io/flipt/tree/643ccc1d541e6a1e79cfeeb548df6b53a09e9914/internal/server/audit) describes how you can implement your own audit event sink.

We hope you found this post useful. As always, you can find us on [GitHub](https://github.com/flipt-io), [Discord](https://flipt.io/discord), [Twitter](https://twitter.com/flipt_io), or [Mastodon](https://hachyderm.io/@flipt).


Audit Events for Flipt


<BlogImage
  src="/dagger/embracing-dagger.png"
  alt="Flipt loves Dagger."
  width={2400}
  height={1350}
>
  <span>
    Background by [Paul Volkmer](https://unsplash.com/@laup) on Unsplash
  </span>
</BlogImage>

In a [previous blog post](/generating-the-flipt-go-sdk), I briefly mentioned our adoption of Dagger.
I wanted to take a moment to introduce Dagger and explain how we’re slowly but surely adapting our build, test (and one-day release) pipeline to leverage it.

> Dagger is a programmable CI/CD engine that runs your pipelines in containers.

[Dagger](https://dagger.io) lets you describe your build, test and release pipelines as code, often alongside the application you’re developing it around. They support an expressive set of SDKs in multiple programming languages. At Flipt we’re using Dagger's (you guessed it) [Go SDK](https://github.com/dagger/dagger/tree/main/sdk/go).

Perhaps unsurprisingly, in Dagger, almost everything is a container. I say unsurprisingly since a lot of the team also gave us Docker, the technology which put Linux containers truly on the map. Each of the Dagger SDKs exposes a builder pattern interface with the primary function of describing a graph of instructions. These instructions are remarkably similar to those found in a Dockerfile, which is not a coincidence. These instructions also produce layers in resulting OCI images, which can then be published or exported from your Dagger pipeline.

I don’t know about you, but I do love a DAG and so does Dagger. The DAG (Directed Acyclic Graph) is intrinsic to what makes it such a neat solution. No points for guessing where they got their name.

<BlogImage
  src="/dagger/dag.svg"
  alt="A DAG"
  width={509}
  height={144}
  caption="A DAG"
/>

A DAG is a set of vertices (the boxes) and edges (the arrows). The edges express dependencies from one vertex to the next. Before I can copy my Flipt binary into an appropriate location for execution, I first need to build it from the source. In order to run `go build` I first need to install Go.

DAGs allow us to express the dependent order in which operations must occur. Where this differs from a plain old sequential list, is that each vertex (entry) can have more than one (potentially) independent inputs. When you describe your workflow in this way, an execution engine (such as Dagger) can leverage the fact that fully _independent_ operations can be executed concurrently. If you’re running your workflow on a multicore machine you may even see these operations run in parallel.

In the rough illustration above, our Go and Node components can be built independently from one another. At this stage there are no interdependencies, so they can be run concurrently. Once both operations have been completed, we bring the results together.

### How does it work?

Dagger itself is comprised of a server and client architecture. The server (known as the engine) runs in a container on a target Docker host. Consumers (like Flipt) use one of the Dagger-generated and managed SDKs in one of the supported languages. These SDKs managed to initialize the Dagger engine container for you, as well making all the GraphQL API calls. Dagger has chosen to embrace GraphQL, which is a particularly good fit when you’re talking in directed acyclic graphs.

Having a GraphQL API means consumers are not strictly bound to only the SDKs developed by the Dagger team. Anyone can take the GraphQL docs and bake their own client in the language of their choice. This is similar to how we at Flipt use protobuf and gRPC and generate our managed clients from these definitions.

<StarCTA />

Under the bonnet Dagger leverages [Buildkit](https://github.com/moby/buildkit/). It was originally created by [Tõnis Tiigi](https://github.com/tonistiigi) and now lives under the Moby (Docker) brand. It powers most of `docker buildx`, where Dockerfile’s are actually transpiled into `llb` instructions (A Buildkit structure for describing DAGs) and then _solved_ (evaluated) by a Buildkit daemon.

If you’re familiar with the concept of `git` and its _plumbing_ vs _porcelain_ commands, I would liken the same to Dagger compared with Buildkit. Buildkit is the plumbing and Dagger is the porcelain. Buildkit exposes a fine-grained API for solving DAGs of instructions, which produce OCI images and layers. Dagger is a fast-evolving, developer-friendly layer on top. A layer focussed on solving the higher-level problem of expressing build, test and deployment pipelines in code.

I could talk about these pieces of tech all day, but for now, here is how we’re leveraging them at Flipt to build and ship quickly with confidence.

## Flipt’s Build, Test, Release Pipeline

<BlogImage
  src="/dagger/build-test-deploy.png"
  alt="The classic build, test, and release pipeline"
  width={2570}
  height={694}
  caption="Build, Test, and Release"
/>

Using Dagger we’re developing the classic build, test, and release sequence.

### Build Phase

The Build phase involves sourcing all the raw materials and putting the puzzle together. Effectively we perform the following five-step process:

1. Establish a base image containing all the build and test tools necessary.
2. Fetch all of our source code from GitHub.
3. Download all the Go and Javascript libraries we need.
4. Build the thing(s).
5. Copy the resulting Flipt binary into its own image with only the necessary _runtime_ dependencies.

If you’ve had experience writing a `Dockerfile` in the past, you may be familiar with some tricks folks employ to make better use of the Docker build cache. For example, with Go you might first add just your `go.mod` and `go.sum` files to your container before the rest of your application. Next, you will likely run `go mod download` in order to populate the module cache. Once this is complete then you add the rest of the application source and perform the build. You might also employ Docker's newer `RUN` cache features and mount a cache volume.

```docker
FROM golang:1.20

WORKDIR /src

ADD go.mod .
ADD go.sum .

RUN --mount=type=cache,target=/go/pkg/mod go mod download

ADD . .

RUN --mount=type=cache,target=/go/pkg/mod go build ./...
```

The same can be achieved with Dagger and the experience is familiar. The following is an illustration of using the Go SDK in Dagger to layer our graph to make the best use of caching for Go applications.

```go
func GoApp(ctx context.Context, client *dagger.Client) (*dagger.Container) {
	deps := client.Host().Directory(".", dagger.HostDirectoryOpts{
		Includes: []string{
			"./go.mod",
			"./go.sum",
		},
	})

	return client.Container().
		From("golang:1.20").
		WithWorkDir("/src").
		WithMountedDirectory("/src", deps).
		WithMountedCache("/go/pkg/mod", client.CacheVolume("go-mod-cache")).
		WithExec([]string{"go", "mod", "download"}).
		WithMountedDirectory("/src", client.Host().Directory("."))
		WithExec([]string{"go", "build", "."})
}
```

Our pipeline for building Flipt ultimately leverages this same process, which is to use layers effectively and take advantage of the engine's layer caching, and mounted cache volumes for finer control ensuring fetched or computed assets are preserved between builds. As with Go, the same applies for Node and its dependencies. First add the dependency manifests (`package.json`), install the dependencies and then add the source code.

Beyond this, we have some other details which are a little more specific to how we build Flipt:

1. The Flipt repo is a Go multi-module repository and uses a [Go workspace](https://go.dev/doc/tutorial/workspaces) to tie it all together. We read the contents of the `go.work` file to identify the multiple sets of `go.mod` and `go.sum` file pairs through the repository and ensure they’re all added before downloading dependencies.
2. `client.CacheVolume(...)` in Dagger takes a string to identify the cache volumes key. We build this key based on a hash of the `go.sum` contents. This invalidates our cache each time our sum file changes. The idea is to keep our cache fresh over time and stop it from growing indefinitely.
3. We initially exclude adding the `ui` directory containing our frontend code from the Go base. Instead, we mount this directory into its own container based on a `node` image and produce a static directory containing all of our frontend assets. This resulting static assets directory is copied into the base Go container and we use Go’s embedding to bake this into the binary.

This process is defined in the function we call `Base(...)` and the result is a `*dagger.Container`. Once all the dependencies have been built and added appropriately, we build the `flipt` binary into a well-known path inside the last layer of the container.

There is another function called `Flipt(...)` which takes the resulting container from `Base(...)` and copies the `flipt` binary into a fresh, empty `alpine` Linux image. This image is prepared with just the runtime dependencies needed for Flipt and gives us a thin distributable image for release.

### Test Phase

There are two important artifacts out of the Build phase which become inputs into our Test phase. The first is the base image (the result of steps 1, 2 and 3) with all the build and test tooling and source code. The second is the distributable Flipt OCI image (the thinner distributable image based on `alpine`).

In the testing phase, we first use the base image again to run our suite of Go unit tests. We run our unit test suite against each of the databases Flipt supports (SQLite, Postgres, MySQL and CockroachDB). The unit test suite requires Go to be installed, along with access to all the source code, which is why we use the base image.

The release of Dagger `v0.4.0` came with support for service containers. If you’re familiar with `docker-compose` and how it supports multiple containers being networked together, then this feature will be familiar. It supports the ability to run containers which are used as dependencies for your workflows as network addressable services. For example, a backing database, or your very own service that you want to test over the network. This feature was crucial for connecting the various databases Flipt supports during unit testing. It also allowed us to further build out our integration testing pipeline.

In a [previous blog post](/generating-the-flipt-go-sdk), we discussed building our new Go SDK. In order to gain confidence in Flipt, its API and our SDK, we leverage the full power Dagger gives us with defining our pipelines in Go.

We first defined a suite of test cases using the Go SDK, which exercises a common set of Flipt operations: create a namespace, create a flag, define its variants and segments, evaluate it, etc. The core functional requirements of Flipt as a feature flagging system.

Given how we built our SDK to abstract away details such as the transport used and authentication, we could then easily extract those (non-functional requirement) details as arguments for running our test suite. This means that we could run our suite against the HTTP or gRPC API with or without authentication enabled. With the introduction of namespaces in our [latest release](https://github.com/flipt-io/flipt/releases/tag/v1.20.0), we further enhanced the suite to run across a configurable namespace. Now that our test suite can be run in a number of configurations, we have an n-dimensional matrix of possibilities.

<BlogImage
  src="/dagger/test-matrix.png"
  alt="Our test matrix"
  width={444}
  height={394}
  caption="Our test matrix"
/>

By simply using a set of nested loops in Go, we have a tidy way of expressing the multitude of possible ways Flipt could be run, giving us confidence that we’re not breaking something each time we add or change some functionality. Couple this with the ability to execute each possible configuration combination in a goroutine and we have a fast, concurrent way of executing the many permutations.

```go
func TestAll(base, flipt *dagger.Container) error {
	var g errgroup.Group
	for _, auth := range []bool{true, false} {
		for _, namespace := range []string{"", "some-namespace"} {
			for _, transport := range []string{"http", "grpc"} {
				g.Go(testFlipt(base, flipt, auth, namespace, transport))
			}
		}
	}
	return g.Wait()
}
```

Our pipeline is now fairly large and relatively complex. Lots of layers and concurrent operations in a web of directed edges and vertices are used to describe both build and test functionality. As with all big and complex CI/CD pipelines, they can start to slow down the more you add. This is where the advanced caching capabilities of Buildkit and Dagger come into play.

The beauty of the DAG in the Buildkit world is that each computed vertex (node) in the graph and its outputs are stored in content-addressable storage. Buildkit takes advantage of when vertices whose definitions and their inputs do not change and avoids recomputing any of these outputs. Instead, it retrieves previously computed results and uses those instead.

Locally, we have seen a full 10-minute end-to-end Flipt build pipeline with subsequent integration suite go from 10 minutes with an empty cache to **17 seconds** on a subsequent run. Flipt currently references 3.2GB **(!)** of Go module dependencies. When you compile Flipt with an empty build cache, a fresh run will produce 1GB of build-cache entries. All this takes a considerable amount of time to fetch and compile. Dagger makes a lot of this work trivial to compute once and reuse on subsequent runs.

Buildkit has the functionality to export its cache storage to a number of remote backends so that you can persist all the hard work done computing layers and cache volumes in CI/CD environments and reuse them between separate triggers of your pipeline. Dagger today (`v0.6.0`) does not expose a stable API for this feature. However, the team are hard at work sweating the details so we won't have to. They’re working hard to find a DX that works for many. At Flipt we’re looking forward to when this lands, as we expect the gains in CI will be very noticeable.

### Release Stage

Once Flipt is packaged and tested sufficiently, it is ready for release. Today we still use the excellent [GoReleaser](https://goreleaser.com/) project to package and ship the final Docker images and binaries to their various destinations. We do suspect though, that this will move to be more Dagger driven as time goes on, ultimately publishing the artifacts produced by the pipelines that we have developed using Dagger.

All the code to define our Dagger-based pipeline can be found in the [build](https://github.com/flipt-io/flipt/tree/main/build) directory of the Flipt repository. We use [mage](https://magefile.org/) both in the root of the Flipt repository and in our Dagger pipeline code as the CLI wrapper. We’re big fans of `mage` and so is Dagger. For those wanting a `make`-like interface which is defined in Go, it is a nice intersection of the two. It also marries nicely with all of our pipeline code being defined using the Dagger Go SDK.

### Dagger All The Way

Dagger is shaping up to be the solution I always wanted for CI/CD. Call me strange, but I am actually excited to develop and maintain these pipelines and I am hopeful it is going to be a sharp tool in our belts here at Flipt.

If you’re interested in Dagger then I highly recommend checking out their [site](https://dagger.io/), [GitHub project](https://github.com/dagger/dagger) and in particular [Discord](https://discord.gg/dagger-io).

Equally, we really want to understand your needs in building and shipping software here at [Flipt](/). So please do come and stop by our [Discord](https://flipt.io/discord) and say hi too.


Dagger, a ❤️  story


<BlogImage
  src="/go-sdk/flipt-sdk.png"
  alt="Flipt's Go SDK"
  width={2400}
  height={1350}
>
  <span>
    Photo by [Lenny Kuhne](https://unsplash.com/@lennykuhne) on Unsplash
  </span>
</BlogImage>

# Generating the Flipt Go SDK

This blog post describes the journey we took to build our new Go SDK, and how we developed our `protoc` plugin in Go to achieve that. It first explores our motivations and then gets into some of the details of how we built it and the tools that go into the automation around it.

We're releasing the new Go SDK today! You should be able to fetch it using `go get go.flipt.io/flipt/sdk/go`.

## Getting Serious About Clients

Recently, we decided to revise and improve our API client offering, starting with our support for Go.

Go is the core language used to develop Flipt's backend service. It is something the team has a long history with, so this was an obvious place for us to start.

With the [recent addition of different authentication methods](/enhancing-authentication#kubernetes-service-account-authentication) for Flipt, we wondered how we could better support integrating these authentication mechanisms into clients.

Our [documentation](https://www.flipt.io/docs/authentication/using-tokens) explains how to manually wire static client tokens into a gRPC or HTTP client, however, we now support authenticating using Kubernetes service-account tokens. Doing so is a more involved process, with an extra API handshake in the middle, so this is something we ideally want our clients to do for you with minimal configuration.

## Status Quo

Flipt’s core service and APIs are defined using [gRPC](https://grpc.io/) and protobuf. This allows us to define our APIs as a set of RPCs and messages in the [proto](https://protobuf.dev/programming-guides/proto3/) language and then use a set of `protoc` tools and plugins to scaffold domain models, servers and clients in different languages. This is a superpower for us. It allows us to reach other programming ecosystems and languages without prior knowledge or experience in them, opening Flipt up to a wider audience.

However, gRPC alone is not for everyone or every ecosystem. While it has a big reach, not every language is catered to equally. In particular, the browser hasn’t had the best support so we also wanted an equivalent HTTP API. The API should also be understandable by developers, allowing them to write their clients using an HTTP library in their language of choice.

This is where [grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway) comes into the mix. Flipt has used this project since the beginning to present an equivalent REST-like API using JSON as the message serialization format. The grpc-gateway tooling supports server generation (not client) in Go.

This is where we were until very recently. We have a gRPC protobuf generated client for the Go community, with no additional bells or whistles, just a thin shim around the Flipt gRPC endpoints and no support for invoking the HTTP API in Go. While we’re uncertain of the use cases, it seemed a shame to mandate gRPC when the [HTTP API](https://www.flipt.io/docs/reference/overview) is also available.

## Our Goals

- A single consistent SDK API, which abstracts the details of the transport from the caller. While we want our users to have the choice, we don’t think the transport details matter when it comes to consuming the client's functionality in your application code.
- Two implementations of the transport abstraction, one for gRPC and one for HTTP.
- To generate as much of this as possible from the original `proto` definitions. We invested in gRPC and protobuf for the tooling and automation. Having our SDK advance with these definitions will keep us up to date.
- Develop an abstraction around authenticating our SDK, which supports consumers’ needs to access Flipt’s APIs with minimal configuration.

<BlogImage
  src="/go-sdk/Flipt_SDK_Architecture.svg"
  alt="Flipt SDK Architecture"
  width={729}
  height={339}
  caption="Flipt SDK Architecture"
/>

The entry point to our new client is the `SDK` type. This type encapsulates the three sections of the Flipt RPC APIs known as `Flipt` (core API), `Auth` (authentication-related APIs) and `Meta` (server metadata APIs). All of these types are generated from the original `proto` definitions.

These SDK types are (mostly) abstracted away from the transport details. The specifics are hidden behind a set of `Transport` Go interfaces. Each of these interfaces has an equivalent implementation for both `grpc` and `http`.

```go
type Transport interface {
	AuthClient() AuthClient
	FliptClient() flipt.FliptClient
	MetaClient() meta.MetadataServiceClient
}
```

For the most part, the `grpc` transport implementation is identical to that of the `grpc-go` generated client. The packages contain some thin wrapping code to glue our SDK together. The crux of the code has been implemented and the transport interfaces are very close to these concrete `grpc` implementations. This was an intentional decision to minimize effort and generated code.

<StarCTA />

The `http` implementation on the other hand required some more work. For a start, `grpc-gateway` does not ship a generated HTTP Go client, so we had to implement this ourselves. To invoke the API generated by grpc-gateway, we need to produce code that can adapt our RPC messages into the appropriate HTTP requests. We also need to provide a Go implementation that conforms to the Transport interface definitions derived from gRPC.

The grpc-gateway ecosystem has a collection of additional annotations and metadata which describe how the RPC methods should be represented as HTTP requests, for example:

```yaml
http:
  rules:
	# ...
	- selector: flipt.Flipt.CreateVariant
	   post: /api/v1/flags/{flag_key}/variants
	   body: "*"
```

Our custom `protoc` plugin parses this additional metadata and uses it to generate Go code which can produce an equivalent HTTP request.

```go
func (x *FliptClient) CreateVariant(ctx context.Context, v *flipt.CreateVariantRequest, _ ...grpc.CallOption) (*flipt.Variant, error) {
	var body io.Reader
	var values url.Values
	reqData, err := protojson.Marshal(v)
	if err != nil {
		return nil, err
	}
	body = bytes.NewReader(reqData)
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, x.addr+fmt.Sprintf("/api/v1/flags/%v/variants", v.FlagKey), body)
	if err != nil {
		return nil, err
	}
	req.URL.RawQuery = values.Encode()
	resp, err := x.client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	var output flipt.Variant
	respData, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, err
	}
	if err := checkResponse(resp, respData); err != nil {
		return nil, err
	}
	if err := protojson.Unmarshal(respData, &output); err != nil {
		return nil, err
	}
	return &output, nil
}
```

I don’t suspect we have managed to support the full breadth of protobuf, grpc and grpc-gateway features. However, we believe we have implemented enough to cover the functionality Flipt takes advantage of.

## How does one write a protobuf Go code generator?

[https://pkg.go.dev/google.golang.org/genproto](https://pkg.go.dev/google.golang.org/genproto) is a Go package designed to aid in the development of `protoc` plugins. It is used by `grpc-gateway` and `grpc-go` in order to parse and generate their respective client and server implementations.

<BlogImage src="/go-sdk/protoc.svg" alt="Protoc" width={561} height={157} />

`protoc` is the entry point to all things protobuf generation. It parses and validates your protobuf files and invokes any configured plugins with a request to generate code on `stdin`.

[https://pkg.go.dev/google.golang.org/protobuf/types/pluginpb#CodeGeneratorRequest](https://pkg.go.dev/google.golang.org/protobuf/types/pluginpb#CodeGeneratorRequest)

The message you are provided is a protobuf serialized structure itself. Specifically, it is a `pluginpb.CodeGeneratorRequest`. This contains the parsed definitions of your protobuf files collected by `protoc`.

It is up to the plugin to interpret this request and produce a response. The response is written to `stdout` as a protobuf encoded `pluginpb.CodeGeneratorResponse`. The response contains all the files you have produced and where they should end up (their file names).

The `protogen` package provides a framework which hides much of the hairy details of writing a `protoc` plugin. Simplifying the wiring and providing useful utilities specifically for generating Go code.

There is an excellent [blog post](https://rotemtam.com/2021/03/22/creating-a-protoc-plugin-to-gen-go-code/) by Rotem Tamir, which laid the foundational knowledge we needed to get started on our custom SDK wrapper and HTTP client. If building something with `protogen` interests you, then I highly recommend reading this first.

### Some `protogen` pointers

I won’t repeat all the wonderful tidbits and details in that blog post, however, I do have a couple of extra pointers that I discovered along the way while developing our SDK generators.

Firstly, we use `proto` version `3` and in particular, we use the `optional` keyword on some message fields. This happens to be a feature of protobuf which was introduced after the initial release of version `3`. If you attempt to use `protogen` on `proto` files using the `optional` keyword without first enabling it as a “supported feature” in your plugin, then you will be confronted with a warning each time your plugin is invoked:

```go
Warning: plugin "go-flipt-sdk" does not support required features.
  Feature "proto3 optional" is required by 1 file(s):
    auth/auth.proto
```

It was a little unclear at first, but the trick is to set the optional feature _bit_ on the `*protogen.Plugin.SupportedFeatures`:

```go
protogen.Options{}.Run(func(gen *protogen.Plugin) error {
		gen.SupportedFeatures |= uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
		// ....
}
```

`SupportedFeatures` is a bitmask and the nice, human-readable names for the feature bits can be found at [google.golang.org/protobuf/types/pluginpb](http://google.golang.org/protobuf/types/pluginpb).

Secondly, managing imports in your generated Go files can initially be unclear. It turns out that `protogen` has a really nice feature, which does a lot of heavy lifting with regard to managing the naming and qualification of those names in your generated output:

```go
for _, f := range gen.Files {
  if !f.Generate {
    continue
  }

  filename := string(file.GoPackageName) + ".sdk.gen.go"
  g := gen.NewGeneratedFile(filename, myTargetPackageName)

  // QualifiedGoIdent will handle adding the GoImportPath
  // to the target GeneratedFile (g) in the `import` declaration
  // at the top. It even handles when you reference two packages
  // with the same name by adding a numeric suffix to distinguish them.
  httpClient := g.QualifiedGoIdent(protogen.GoIdent{
    GoImportPath: protogen.GoImportPath("net/http"),
    GoName:       "Client",
  })

  g.P("type MyClient struct {")
  g.P("client *", httpClient)
  g.P("}")
}
```

`protogen.GeneratedFile.QualifiedGoIdent` ensures that the path you’re referencing is correctly imported into the top of the resulting generated Go file. It also handles any imports with duplicate names by adding a numeric suffix (e.g. importing both `text/template` and `html/template` would resolve to `template "text/template"` and `template1 "html/template"`). Then the resulting import path is correctly prepended onto the type when the `GoIdent` produced by this function is passed into `g.P("...", httpClient)`.

The example generator code above would produce something similar to:

```go
package client

import http "net/http"

type MyClient struct {
	client *http.Client
}
```

### Putting it altogether

So Flipt now has a single `protoc` plugin, which produces three target packages.

The plugin lives [here](https://github.com/flipt-io/flipt/tree/main/internal/cmd/protoc-gen-go-flipt-sdk) and it generates the following three packages into the Flipt repository:

- `sdk/go/http` containing `HTTP` implementations of the `sdk.Transport` interfaces.
- `sdk/go/grpc` contained `gRPC` implementations of the `sdk.Transport` interfaces.
- `sdk/go` the core of our Go SDK.

To bring all of this together, we use a tool called [buf](https://buf.build/) to orchestrate the invocations of the `protoc` toolchain. The protobuf toolchain is a fiddly beast and `buf` greatly simplifies this process for us.

We have a single addition to our `buf.gen.yaml` which drives this tool. It looks like this:

```yaml
plugins:
  # ...
  - name: go-flipt-sdk
    out: sdk/go
    opt:
      - paths=source_relative
    strategy: all
```

Each of the entries under `plugins` is treated as a `protoc` plugin. These will be passed on invocations of the `protoc` tool by `buf`. `protoc` knows when it sees the name `go-flipt-sdk` to prepend `protoc-gen-` onto the front and look that binary in your `PATH`. In our case, it looks for `protoc-gen-go-flipt-sdk`.

## Where next?

We’re hoping this automation keeps us on a good footing with each change to Flipt’s API, allowing us to generate a client in lockstep with the server API, which has support for all the RPC definitions we create.

Beyond just generating the client we plan to add support for the Kubernetes authentication method, reducing that down to some configuration with sensible defaults.

Finally, we have begun the work to build out a new integration test suite to ensure our SDK and Flipt are behaving as we intend. Our next blog post on this topic will explore how we’re building out a new CI pipeline using [Dagger](https://dagger.io). Dagger is enabling us to define a pipeline we can execute anywhere, which is both fast and reproducible. We’re very excited about the future of Dagger and can’t wait to talk about how we plan to leverage it in the Flipt build process.

We hope you’ll find this new SDK useful, and we’re happy to answer any questions that you may have in our [community Discord](https://flipt.io/discord) or via GitHub issues/discussions.

We welcome all feedback on how to improve the SDK or Flipt as a whole, so feel free to reach out! You can find us on [GitHub](https://github.com/flipt-io), [Twitter](https://twitter.com/flipt_io), [Mastodon](https://hachyderm.io/@flipt), and [Discord](https://flipt.io/discord).


Generating the Flipt Go SDK


<BlogImage
  src="/hi-yoofi/yoofi.png"
  alt="Hello, Yoofi!"
  width={2400}
  height={1350}
>
  <span>
    Photo by [Dejan Zakic](https://unsplash.com/@dejanzakic) on Unsplash
  </span>
</BlogImage>

We are excited to announce a new addition to the Flipt team, [Yoofi Quansah](https://www.linkedin.com/in/redblacktree/) - a Senior Software Engineer with 6+ years of experience and a love of developer tooling. With his impressive background and skills, we are confident that he will play a significant role in advancing our mission of helping engineers everywhere deploy software to production without fear.

Here's what Yoofi had to say about his background and decision to join the Flipt team:

> Hello 👋, my name is Yoofi Quansah. I've had quite the journey to finally find my way here at Flipt, but let me not bore you with details you probably don’t want to know and get to the fun stuff!
>
> Before writing code, I was actually 99% positive I was going to make it to the NFL (as we all were). I am writing this blog post now, so you know that didn't happen 🙂, but I am very happy with how my life has turned out so far.
> I started my career out building simple developer tools over at Honey. After I got over the hump of imposter syndrome and really got into my work, we started to build some really cool tools on the team. These tools included our own feature flagging service (foreshadowing), deployment platforms, and testing platforms to name a few.
> Everything was cool right? Not quite. I realized that even though we were building these cool tools, we were not really getting much recognition for it, and even worse the engineers were relatively slow to adapt. It was quite draining, to say the least.
>
> Enter July 2019... My life changed when I went to Gophercon.
> I found people who were building developer tools and directly interacting with customers with the tools they built. Wow. It was at that point that I knew I wanted to write Go, and work with a company that interacted directly with developers as their customers. After all, it is pretty nice to see the value of the products that you build for others to use directly.
> The result of all this was me leaving my job at Honey, and joining InfluxData in August of 2020, where I met [George](/welcome-george-macrorie)!
>
> InfluxData was an incredible experience. I met tons of smart people, which allowed me to hone my craft as an engineer. It also allowed me to realize exactly what I wanted out of my career, which was to build incredible features for engineers to use. Most of my time there was taken up building the next generation of the query engine, which solved a wealth of problems we were facing with the old one.
> George introduced me to the grand idea of Flipt, all Mark has already done already, and the future outlook/potential roadmap in early 2023. Even though I was quite happy at InfluxData, I fell in love with the challenge, and all that is being hoped to accomplish.
>
> Combined with my previous experience helping out building a feature flagging service at Honey, and a whole lot of issues I ran into with feature flags both at Honey and InfluxData, I believe there is a bright future for this critical piece of software development and delivery.
> I am happy to be here, and hack on!

Welcome Yoofi, we are lucky to have you! 🚀


Meet Yoofi Quansah


<BlogImage
  src="/enhancing-authentication/enhancing-authentication.png"
  alt="Enhancing Flipt Authentication"
  width={2400}
  height={1350}
>
  <span>
    Photo by [Roon van der Meer](https://unsplash.com/@roonz_nl) on Unsplash
  </span>
</BlogImage>

In the previous few posts, we went over the addition of one of our most asked-for features: User Authentication support. Just when you thought that might be the end of our authentication story, we improved it even more by adding three more authentication-related features:

1. API token configuration via the UI
1. Configurable bootstrap token
1. Kubernetes Service Account authentication

In this post we'll briefly go over each feature, discussing its use case, why it's an important addition to our overall auth story, as well as how to get started.

## API Token Configuration

We've had basic token authentication support for authenticating your services with Flipt since v1.15. Admittedly though, it was somewhat incomplete as you could only manage those tokens via the [API](https://www.flipt.io/docs/reference/authentication/create-token).

In this [latest release](https://github.com/flipt-io/flipt/releases/tag/v1.19.0), we added the ability to create and delete static tokens in the UI as well. This allows anyone authenticated with Flipt to see which tokens have been issued and when they expire. It also allows users to create and delete tokens as needed without having to write code to do so.

Enabling and using static token authentication is a good first step when it comes to securing the communication between Flipt and your own services. Making the management of access tokens as user-friendly as possible was one of the major acceptance criteria for this new functionality.

This feature also necessitated a new 'Settings' section of the application which we'll populate with more controls in the future.

<BlogImageGallery
  images={[
    {
      src: "/enhancing-authentication/static-tokens/list.png",
      alt: "Settings Screen with Static Tokens View",
    },
    {
      src: "/enhancing-authentication/static-tokens/new.png",
      alt: "Creating a New Token",
    },
    {
      src: "/enhancing-authentication/static-tokens/expiry.png",
      alt: "Adding an Expiration to a Token",
    },
    {
      src: "/enhancing-authentication/static-tokens/created.png",
      alt: "Token Created",
    },
  ]}
/>

After creating a token via the UI, you can easily copy it to your clipboard and store it in a secure location as its value will no longer be accessible from Flipt itself.

## Configurable Bootstrap Token

Another improvement that we added in this release is the ability to provide a configurable bootstrap token when first enabling token authentication. Since v1.15, if an operator enabled token authentication for the first time, we would create what we call an initial bootstrap token on their behalf, logging it to `STDOUT` to be captured and used.

```console
access token created	{"server": "grpc", "client_token": "ks8gyZNdU7jmB92kAIYOp4MEXTcCJbX1TdXJM01VKCs="}
```

This was necessary because all requests to the Flipt API would fail as soon as you enabled and required token authentication.

This meant, however, that there wasn't a great programmatic way to run Flipt with token authentication enabled for the first time, such as in the case of Kubernetes. Flipt operators would have to either:

1. Start Flipt with authentication enabled but not required, create a token via the API, and then restart Flipt with authentication required.
1. Start Flipt with both authentication enabled and required and try to capture the created bootstrap token from our logs.

This also presented a similar challenge when trying to run Flipt in CI, such as we do for our own integration tests, as there was no easy way to set up authentication and then restart Flipt in a test pipeline.

In this release, we added the ability for the user to optionally specify their own known bootstrap token as well as to configure it with an expiration.

<BlogImage
  src="/enhancing-authentication/bootstrap-token-expiration.png"
  alt="Bootstrap token created with expiration"
  width={2442}
  height={490}
  caption="Bootstrap token created with expiration"
  className="shadow-md"
/>

Here's an example of how to configure a bootstrap token with a known value and an expiration of 24 hours:

```yaml
authentication:
  methods:
    token:
      enabled: true
      bootstrap:
        token: "s3cr3t!"
        expiration: 24h
```

In our opinion, this provides the best of both worlds by allowing for the initialization of an authentication token with a potential known value that can expire automatically. This gives operators the ability to enable authentication from the get-go, use a known token to create more tokens, then allow the known token to quietly expire to prevent unwanted access.

More information on how to configure this feature can be found in our [documentation](https://www.flipt.io/docs/configuration/authentication/#token-authentication).

## Kubernetes Service Account Authentication

In most cases, static tokens are a manageable approach for getting services talking with Flipt. For security reasons, you may set an expiry on your tokens and periodically rotate them with new ones. Given you have more than one type or replica of your application(s) communicating with Flipt, you may even choose to create multiple tokens. It may be one per application type or even as many as one per replica.

For those deploying Flipt into Kubernetes environments, we have added the new `kubernetes` authentication method.
This method takes advantage of the [service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) mechanisms made available in Kubernetes.

By default Kubernetes periodically writes a service account token (encoded as a JWT) into a well-known path inside each pod's filesystem.
Service account tokens can be used to communicate and identify the caller with the Kubernetes API. With Flipt [v1.19](https://github.com/flipt-io/flipt/releases/tag/v1.19.0) you can exchange this service account token for a Flipt client token via Flipt's API.

A small amount of client-side glue code can keep your applications authenticated without having to manually create, rotate or distribute tokens.

Here's an example using `curl` and `jq` to exchange a Kubernetes service account token for a Flipt client token:

```bash
curl -s -X POST http://flipt:8080/auth/v1/method/kubernetes/serviceaccount \
  --data "{\"service_account_token\":\"$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\"}" | \
  jq .
{
  "clientToken": "pKkaEik40Nu4lJ7O37l92MNyyD38U8UlaagSmAfJoS0=",
  "authentication": {
    "id": "f191babc-57b8-4856-89b1-f324941403d7",
    "method": "METHOD_KUBERNETES",
    "expiresAt": "2024-02-20T13:39:14Z",
    "createdAt": "2023-02-20T14:11:28.962841Z",
    "updatedAt": "2023-02-20T14:11:28.962841Z",
    "metadata": {
      "io.flipt.auth.k8s.namespace": "default",
      "io.flipt.auth.k8s.pod.name": "someservice-586bfb5b6b-fmh8g",
      "io.flipt.auth.k8s.pod.uid": "b5217947-aeac-4b35-afd3-23f50d63eae9",
      "io.flipt.auth.k8s.serviceaccount.name": "default",
      "io.flipt.auth.k8s.serviceaccount.uid": "8aeb28ad-66f0-4884-bafc-e606e5eda149"
    }
  }
}
```

Enabling this method means Flipt implicitly trusts other services deployed into the same cluster.

While currently leveraging this new method will require some application code on the developers' part, we're actively working on a way to provide out-of-the-box support for our official clients.

As always, our [documentation](https://www.flipt.io/docs/configuration/authentication) has more information on how to configure these features.

Please let us know what you think of these new features and if you have any questions or feedback, feel free to reach out to us on [Discord](https://flipt.io/discord), [Twitter](https://twitter.com/flipt_io), or [Mastodon](https://hachyderm.io/@flipt)!


Enhancing Authentication Support


<BlogImage
  src="/new-look-and-oidc/flipt-v1.18.0.png"
  alt="Flipt v1.18.0"
  width={2400}
  height={1350}
>
  <span>Photo by [Jr Korpa](https://unsplash.com/@jrkorpa) on Unsplash</span>
</BlogImage>

I'm happy to announce that there's a [new version of Flipt, v1.18.0](https://github.com/flipt-io/flipt/releases/tag/v1.18.0) which has a few important changes.

- The whole UI has been rewritten in [ReactJS](https://reactjs.org/) and styled with [TailwindCSS](https://tailwindcss.com/). Technically this happened in v1.17, but v1.18 has a few new features that we want to highlight, so we decided to wait and officially announce both at the same time.
- New authentication options — You can now authenticate into Flipt using Google, GitLab, Auth0, Dex, or any [OpenID Connect](https://openid.net/connect/)-compliant service.

## Refreshed User Interface

In the [previous post](/embedding-react-in-go), we went over the technical challenges and decisions we made to integrate our new UI, written in ReactJS, with the existing Flipt backend, written in Go. In this post, we'll quickly cover the why behind this initiative, the result of this effort, and explain how our new UI helps us to improve your experience as an end user of Flipt.

Our previous UI was written in [VueJS](https://vuejs.org/) and it had started showing its age. We had tried to keep it up to date as far as library and dependency versions, however, there was no getting around the fact that it had been written for our needs a few years ago and wasn't as flexible as we needed it to be going forward.

First off, we were using VueJS 2, which was a major version behind the current version of VueJS, 3. While it might not have been too difficult to update our application to the latest version of Vue, we theorized that it would end up requiring a lot more work than if we started fresh by rethinking the user experience and then picking a new framework.

In fact, the component library that we were using in our Vue2 app, [Buefy](https://buefy.org/), has [explicitly stated](https://github.com/buefy/buefy/issues/2505#issuecomment-997000720) that they would not support Vue3 going forward. While this is completely understandable and not a knock on Buefy, it was a concern for us as it meant we were 'stuck' on Vue2.

The second major issue with our front-end application was that it was written in plain JavaScript, so we weren't able to take advantage of all the nice type safety and tooling that you get with TypeScript. This ended up being another supporting reason as to why made the switch from VueJS to React since we knew we wanted to move to TypeScript and would have to effectively re-write the entire application anyways.

The final reason we decided to start from scratch with the UI was that we wanted to create a new, aesthetically pleasing look while enabling our small team to continue to work effectively and be able to ship new features for the front end. Thank goodness libraries such as TailwindCSS exist, allowing even us non-designers to quickly create an attractive UI that we could be proud of.

We were able to accomplish all of these things by switching over from VueJS to React, and the end result was a UI that looked much better and was easier for our team members to develop and maintain. We also decided that React would be a better choice for our application long term because of React's popularity, which makes it easier to find contributors and team members who enjoy working with it.

While we probably could have tackled some of these concerns by better componentizing the existing UI, or upgrading to Vue3 without Buefy, in the end, I think we made the right choice.

The result of our work is a brand-new UI that is responsive, easy to navigate, and looks great on all devices. Here are a few screenshots of the UI in action:

<BlogImageGallery
  images={[
    {
      src: `/new-look-and-oidc/screenshots/flags.png`,
      alt: "Flag List",
    },
    {
      src: "/new-look-and-oidc/screenshots/variants.png",
      alt: "Add Variant",
    },
    {
      src: "/new-look-and-oidc/screenshots/segments.png",
      alt: "Segment List",
    },
    {
      src: "/new-look-and-oidc/screenshots/distributions.png",
      alt: "Configure Distributions",
    },
    {
      src: "/new-look-and-oidc/screenshots/evaluation_console.png",
      alt: "Evaluation Console",
    },
  ]}
/>

## New Authentication Options

We’re excited to announce that Flipt now supports additional authentication options besides the static token authentication that we added in [v1.15.0](/authenticating-flipt). This new, session-enabled, authentication is powered by the [OIDC (Open ID Connect)](https://openid.net/connect/) protocol, which allows you to use your existing identity providers like Google, GitLab, Auth0, [Dex](https://dexidp.io/), or Okta to log in to Flipt.

In the past we didn't have a great answer for how to secure the Flipt user interface from the outside world, other than to simply:

- Not expose it by putting it behind a VPN
- Put a reverse proxy in front and delegate the authentication to the proxy.

As noted above, in v1.15.0 we added static token support so that you could secure your integrating service-to-service communication with Flipt, but this still did not solve the problem of an end user wanting to authenticate with the UI.

Since OIDC is a popular standard for authentication, enabling it allows Flipt to support a variety of different identity providers using just one login flow, which makes it easier for you to secure your Flipt UI from the outside world.

So what does this mean for end users? If you configure an OIDC provider, Flipt will present it as a login option when you visit your Flipt dashboard. Once the end user authenticates with the provider, they will be returned to the Flipt dashboard. Flipt will establish a client token and set it in a protected cookie. The token has a configurable lifetime and will be rejected once that lifetime expires.

<BlogImage
  src="/new-look-and-oidc/login.png"
  alt="Flipt Login Screen"
  width={441}
  height={444}
  caption="Login Page with OIDC Providers"
  className="shadow-md"
/>

There are several new configuration settings that you can use to set up and customize your OIDC integration, and they are thoroughly documented in our updated docs. You can read more about them here:

- [OIDC Configuration Settings](https://www.flipt.io/docs/configuration/authentication#method-oidc)
- [OIDC Method Overview](https://www.flipt.io/docs/authentication/methods#oidc)

## Wrap Up

In future updates we'll be adding support for other non-OIDC-based authentication such as [GitHub authentication](https://docs.github.com/en/developers/apps/building-oauth-apps/authorizing-oauth-apps) (via OAuth2) as well as providing UI functionality to manage your static tokens and more.

We hope you’ll find this update useful, and we’re happy to answer any questions that you may have in our [community Discord](https://flipt.io/discord) or via GitHub issues/discussions.

We also welcome all feedback on how to improve Flipt as a whole, so feel free to reach out! You can find us on [GitHub](https://github.com/flipt-io), [Twitter](https://twitter.com/flipt_io), [Mastodon](https://hachyderm.io/@flipt), and [Discord](https://flipt.io/discord).


New Look + Authentication Options


<BlogImage
  src="/embedding-react-in-go/react-go.png"
  alt="Flipt React + Go"
  width={2416}
  height={1366}
>
  <span>
    Photo by [Jason Leung](https://unsplash.com/@ninjason) on Unsplash
  </span>
</BlogImage>

We’re not far off the 1.17 release of Flipt. It comes with a fresh coat of paint and some foundational changes to the Flipt user interface. We wanted to take a moment and share part of the engineering journey taken to get to this release.

## TL;DR

- We’re moving from Vue to React 🎉
- We experimented with Next.js 🧪
- We struggled with client-side routing 🗺️
- We settled on plain-old React + hash-based routing ⚛️

As we neared the end of 2022, we were settling new authentication features we had been building. The ability to support login in the UI was a primary motivator for the backend changes we were making, so naturally, we were closing in on building login functionality in the frontend. Since we had to tackle some reasonably sized frontend changes anyway, we took a moment to reconsider the state of our frontend architecture.

When Mark first laid the foundations of the current UI back in February 2019, Vue.js presented a quick and easy way to do so. Particularly, in such a way that could be easily embedded into Flipt.
Flipt is a single binary built primarily in Go, and it serves both the HTTP and gRPC protocols.

The UI is served over HTTP directly from the binary using Go’s fantastic [http.FileServer](https://pkg.go.dev/net/http#FileServer) implementation from the standard library. Sometime around February 2021 was the Go 1.16 release, which came with [native embedding](https://github.com/golang/go/issues/41191) of files into Go binaries. Flipt quickly took advantage of this feature for bundling the static UI assets directly into the binary. Go has a handy wrapping function that adapts the `embed.FS` into the required `http.FileSystem` interface to integrate it seamlessly into an `http.Server`.
You can find the current (`v1.16.0`) embed directive in Flipt [here](https://github.com/flipt-io/flipt/blob/9e0ba7d839ef452526815f71ed6e1207d6aa4997/ui/embed.go#L8-L9).

<BlogImage
  src="/embedding-react-in-go/ui-build-process.png"
  alt="UI to Go embedding build process"
  width={1646}
  height={503}
  caption="UI to Go embedding build process"
/>

Fast-forward back to today, and we're reviewing the future of the UI. It appears that React has dominated the world of frontend frameworks since Flipt inception. As such, we have speculated that React could have a broader appeal for Flipt contributors, so we have decided to take the plunge now before we get any more invested in frontend features.

The frontend is not our natural habitat at Flipt. We’re learning by doing. When we decided to take the first steps to transition from Vue to React, we looked around for what is the current hot stuff on the market.
This is where [Next.js](https://nextjs.org/) caught our attention. We had some initial familiarity with it, as this blog and our landing page is built using it.
Next has plenty of batteries included tooling and how-to material to sink our teeth into, so we got to work spiking a prototype.

It didn’t take long for us to get a prototype using Next.js together. Where things started to get tricky was getting Next.js to fit in our Go binary, where the existing UI currently resided.

## Challenges

### Challenge 1: Bundling our Next.js application into a Go binary

Our first goal was to try to replicate what we had already. Easy right? Last time we just:

1. Ran a build process in the UI project to produce a directory of assets.
2. Embedded those assets using Go’s embed tooling.

This first attempt didn’t take too long to replicate. A little digging around Next’s documentation led us to this page: https://nextjs.org/docs/advanced-features/static-html-export. After building the site with `npm run build`, a subsequent `next export` appeared equivalent to how we produced and consumed our Vue.js application. A little more digging to get the export CLI flags right, and it was as simple as `next export -o $FLIPT/ui/dist`.

Now we have all our assets (HTML, JS, CSS etc) in the correct directory. Without changing anything else, we booted Flipt and took a look. At first glance, everything seemed like it might have just worked.

### Challenge 2: Missing Assets

However, things were not quite what they seemed. Interactions weren’t working as expected. A brief look under the hood (via developer tools), and sure enough, there were a bunch of errors.

<BlogImage
  src="/embedding-react-in-go/console-errors.png"
  alt="Errors in the browser console"
  width={2024}
  height={920}
  caption="Errors in the browser console"
/>

The errors were a little misleading at first. What has this got to do with `X-Content-Type-Options: nosniff`? Why is `text/plain` being returned as the content-type for JS assets by our file server? Popping open the network tab was a little more enlightening.

<BlogImage
  src="/embedding-react-in-go/console-network-requests.png"
  alt="Console network tab requests"
  width={2024}
  height={920}
  caption="Console network tab requests"
/>

These failing assets were returning 404 (Not Found) status codes, but not everything was returning a 404.
Looking at the responses a little closer shined a light on our weird `Content-Type` and “blocked” error messages. Our Go static file server was presenting a generic 404 response page (Content-Type: text/plain) for our missing assets. These assets were being requested via `script` tags, and their security mechanisms were kicking in when they got back a content-type they weren’t expecting.

Long-story short, our Go file server was not serving some subset of our assets.
A quick check of the asset folder we created earlier, and the assets were actually where they should be.

```
➜  tree ui/dist
.
├── 404.html
├── _next
│   ├── E5aRj9aVYvtSy47LN7sYl
│   └── static
│           ├── E5aRj9aVYvtSy47LN7sYl
│           │   ├── _buildManifest.js
│           │   └── _ssgManifest.js
// trimmed output for brevity
```

So, what was it that these particular assets had in common, separate from the assets which were being served successfully?

<BlogImage
  src="/embedding-react-in-go/console-network-requests-ordered.png"
  alt="Console network tab requests ordered"
  width={2056}
  height={950}
  caption="Console network tab requests ordered"
/>

Could it be that they start with an underscore? A bit of digging into Go’s [embed](https://pkg.go.dev/embed#hdr-Directives) standard library turned up just what we were looking for.

```
If a pattern names a directory, all files in the subtree rooted at that directory are embedded (recursively), except that files with names beginning with ‘.’ or ‘_’ are excluded.
…

If a pattern begins with the prefix ‘all:’, then the rule for walking directories is changed to include those files beginning with ‘.’ or ‘_’. For example, ‘all:image’ embeds both ‘image/.tempfile’ and ‘image/dir/.tempfile’.
```

It appears that all we needed was to update our call to the embed directive to the following:

```diff
-// go:embed dist/*
+// go:embed all:dist/*
var UI embed.FS
```

This did the trick. At-least, the errors in the console were gone. All assets were returning status code 200.

### Challenge 3: Right-Click, Open New Tab

It didn’t take long to stumble onto the next set of issues and subsequent can of worms.
When attempting to open a link in a new tab, or visiting it in a new window, we were encountering errors.

Some actions returned 404s, and some listed links to other pages with unexpected content.

<BlogImage
  src="/embedding-react-in-go/segments-page-listing-contents.png"
  alt="Segments page returning directory contents"
  width={2378}
  height={1774}
  caption="Segments page returning directory contents"
/>

What is `[pid].html`? Why are we seeing 404s attempting to create a new flag? I thought this was simply another single-page app? Are we actually navigating to non-existent pages?

Comparing old and new side-by-side, we spotted something we had not considered during the transition. The routing appeared to be different.

<BlogImage
  src="/embedding-react-in-go/current-flipt-ui.png"
  alt="Current Flipt UI Routing Contains Hash"
  width={2040}
  height={1602}
  caption="Current Flipt UI Routing Contains Hash"
/>

We spotted the `#` before the path in our existing Vue.js application. The hash in the URL path (known as a URI fragment) is never sent as part of a server-side request. It is purely there to serve the frontend. When added to a URL, most browsers will attempt to locate an equivalent HTML ID attribute in the page and scroll to that location in the view. However, it has since become popular to leverage the hash value to drive single-page application paths.

In reality, only the root `index.html` (the file server for `/` path) file is being served.

In hindsight, that makes sense. It is a single-page application.
However, in our new application, we have real paths in the URL. These appear to work as expected when you stay within the same window, but the moment you navigate from a new context, we see attempts to serve these paths from the Go file server.

This is where the differences in the transition really started to become clear.
We put on our thinking caps to explore a path forward.

_Unsurprisingly, as the dust settled, some of these problems and solutions started to present themselves in other documentation and blog posts. That was after some failed attempts to Google the problem and come up with our own hacks solutions. Fortunately, we were on the right path._

We explored and prototyped three different paths forward.

1. Update our Go file server to only serve `index.html` for all (HTML asset) paths.
2. Teach our Go file server how to handle Next.js [File-System Routing](https://nextjs.org/docs/routing/introduction).
3. Change back to hash-based routing in our frontend application.

#### Serving `index.html` for all HTML requests

One quick trick (which we later found outlined in [Vue’s documentation](https://v3.router.vuejs.org/guide/essentials/history-mode.html#example-server-configurations) on a similar subject) is to serve the `index.html` page generated by Next, whenever any request for HTML is made.

| path requested | currently rendered | desired page |
| -------------- | ------------------ | ------------ |
| /flags         | /flags.html        | index.html   |
| /segments/one  | /segments/one.html | index.html   |
| /console       | /console.html      | index.html   |

You get the idea.

Though this isn’t the full picture as one small additional change is required.
By default, it appears that the Next.js router doesn’t attempt to reconcile the actual browser path (window.location.path) with the routers' initialization value for the path.
When we render `index.html` for say the `/segments` page, the router initializes with the path `/`.
I suspect our particular use case is something Next.js isn’t trying to cater for.

In order to get around this problem, we employed a quick hack.
On page load, we forcibly update the router's path if it was not as expected via a [`useEffect`](https://reactjs.org/docs/hooks-effect.html).
This causes a little visual hop from the root page to the desired destination... but it works.
If you know of a better way of doing it, come nerd-snipe us in our [Discord](https://flipt.io/discord).

#### Implementing Next.js File-System Routing in Go

This approach was a little heavier on the Go side. The crux of the solution is to support Next.js ability to serve dynamic pages via an on-disk naming convention using square brackets in the filename (This is our friend `[pid].html`).

Firstly, there is one-quick trick we did with `next export` functionality, that saved us some further complexity in our implementation. Set the `trailingSlash` property in the Next configuration to true.
https://nextjs.org/docs/api-reference/next.config.js/trailing-slash

This changes the exported output structure of the static HTML slightly.

- `segments.html` becomes `segments/index.html`
- `[pid].html` becomes `[pid]/index.html`.

This happens to be more congruent with how the Go file server handles particular request paths.
When a path happens to refer to a directory, like `/segments` is in Flipt, Go will render an HTML snippet containing a list of links to the directories contents.
This is what led to the screenshot earlier. Containing the following files:

- [pid].html
- new.html

When that directory happens to contain an `index.html`, it serves that instead.

This gets us a lot of the way there, however, the final challenge is dynamic routes in Next. For example, when requesting the path `/flags/my_flag_key`, the standard Go file server will attempt to open the following two files:

- /flags/my_flag_key
- /flags/my_flag_key/index.html

Neither of these files exists, as the `my_flag_key` refers to a flag key stored within the Flipt backend system. With next.js, what we really want to serve is `/flags/[pid]/index.html`. This page knows how to handle extracting the key from the path and requesting it from the backend.

We need a way to map all dynamically routed paths to their respective dynamic files “on-disk”.
I say “on-disk” in inverted quotes because remember, we embed these files into our Go binary earlier. The assets are not really on-disk. They are exposed via a filesystem abstraction (an interface) in Go.

Go’s [embed.FS](https://pkg.go.dev/embed#FS) is a structure that implements the [fs.FS](https://pkg.go.dev/io/fs#FS) interface.
This interface requires a single `Open(name string) (fs.File, error)` method.

Given that we know and can explore the contents of this directory on initialization of Flipt; we do so to derive the expected dynamic paths. Then we implemented a decorator of the `fs.FS` interface, which decides whether an incoming file path needs to be re-directed and open one of the dynamic paths.

```go
type nextFS struct {
  // we will slot our embed.FS instance in here
  fs.FS

  // prefix tree of potential paths
  tree *node
}

func newNextFS(embedded fs.FS) fs.FS {
  var root *node
  // walk provided filesystem "embedded" to derive
  // potential dynamic paths and build a tree.

  return &nextFS{
    FS:   embedded,
    tree: root,
  }
}

func (*nextFS) Open(name string) (fs.FS, error) {
  var adjustedPath string

  // walk tree in order to replace any dynamic path
  // parametes with the name of the dynamic paths on disk
  // building and adjusted path to the desired file.

  return n.FS.Open(adjustedPath)
}
```

Our first pass implementation of this can be found [here](https://github.com/flipt-io/flipt/blob/88ce30f342075fa216c1b64b76a05b86ae0009cb/internal/cmd/ui.go#L59).
The TL;DR is we built a crude prefix tree based on the paths in the embedded filesystem, with the dynamic files annotated within the tree.

When a requested file matches a dynamic path in the prefix tree, we re-write the path name before handing it on to the embedded `fs.FS` implementation.

Now `/flags/my_flag` serves the contents of `/flags/[pid]`.

#### Embracing Hash Routing

Finally, hash-based routing. This is what our current UI does with Vue.js.
Re-instating this routing capability as the primary approach to client-side routing brings us back in line with how Flipt works today.
It fully encapsulates Flipt as a single-page application, meaning, we can easily embed it into a simple, vanilla file server implementation in Go.
Furthermore, this approach is backward compatible with existing Flipt URLs (without additional changes).

As this potential strategy dawned on us, we also decided to review our decision to use Next.js.
We still feel that adopting React is a worthwhile bet for the future of Flipt, however, we don’t have to use Next.js to do so. In fact, we suspect vanilla React will be more broadly understood than Next.js right now.
Next.js brings plenty of extra opinions, which in hindsight are not necessarily congruent with where Flipt is going.

Finally, React Router has an out-of-the-box implementation of [hash routing](https://reactrouter.com/en/main/routers/create-hash-router) right there for the taking.

#### The Decision

After exploring all of the above, we settled on approach (3), vanilla React and hash-based routing.

_Sorry, perhaps next time Next.js. You’re very cool and shiny. But React is all we require right now._

The product of this choice is no change to the Go side of the equation.
The frontend application involves a simple npm-driven build process, which outputs a “dist” folder containing all the necessary assets.
This folder contains a single HTML file (`index.html`), along with several Javascript and CSS files.
We embed the folder exactly as we had before, using the Go standard library to both embed and serve it.

## Final Words

We hope that moving from Vue to React will attract more contributions from a wider pool of developers.
Flipt is an open-source solution, and we want to build an active community around it.

The new UI will be part of the `v1.17.0` release of Flipt.
We hope you will take a moment to give it a test drive.
Hot on this release's heels will be login functionality in the UI.

We intend to publish another blog post around `v1.18.0` talking more about the journey to this new UI, the new authentication related features and where we’re going next.


Embedding Our New React UI in Go


<BlogImage
  src="/grafana/announcing-dashboards.png"
  alt="Grafana Dashboards"
  width={2400}
  height={1350}
>
  <span>Photo by [Jay R](https://unsplash.com/@justjthings) on Unsplash</span>
</BlogImage>

We recently started developing a collection of Grafana Dashboards to provide visibility
into both the health of your Flipt deployment and the state of the evaluation system.

In [Flipt v1.16.0](https://github.com/flipt-io/flipt/releases/tag/v1.16.0) we have made some alterations to the metrics that we expose. This includes some measurement naming changes, along with the addition of some new ones, specifically around the evaluation engine.

These new dashboards depend on the names and new metrics added in [v1.16.0](https://github.com/flipt-io/flipt/releases/tag/v1.16.0). Upgrading to this version is a requirement before attempting to use these new dashboards.
Metrics are exposed as Prometheus-style metrics. Our dashboards have been developed against a target Prometheus instance. See [configuring metrics](https://www.flipt.io/docs/configuration#metrics) for documentation on configuring metrics in Flipt.

Currently, there are two separate dashboards—one for system overview and the other for flag evaluation state.

Flipt System is the name of the system dashboard. It includes operational details of Flipt, including RPC request throughput, response codes, and latency, as well as details relating to interactions with the backing SQL database.

<BlogImage
  src="/grafana/flipt-system-dash.png"
  alt="Grafana Flipt System Dashboard"
  width={3336}
  height={1416}
  caption="System Dashboard"
/>

Flipt Flag Evaluations dashboard includes details specific to each flag. Selecting a specific flag will present historical details on the resulting matches, match reasons, segments, and values produced.

<BlogImage
  src="/grafana/flipt-flag-dash.png"
  alt="Grafana Flipt Flag Evaluation Dashboard"
  width={3354}
  height={1926}
  caption="Flag Evaluation Dashboard"
/>

The system dashboard is more focused on operators trying to understand the health of a Flipt deployment, whereas the evaluations dashboard is focused on Flipt users who want to understand what Flipt is returning and why. It is critical as an end user of a feature flag system to understand how your flags are being evaluated in a production context so that you can answer questions like "Is my flag being referenced?", "Is it safe to delete this flag yet?" or "How often is variant A returned compared with variant B?".

This is our first pass at building reusable dashboards for Flipt. Our dashboards are open-source and we invite anyone who would like to contribute and improve them. We intend to keep them maintained and preserve a level of compatibility going forward with respect to measurement naming.

Head to our [Grafana dashboards repository](https://github.com/flipt-io/grafana-dashboards) to try them out or contribute.

As always, if you have any questions or feedback, please reach out to us on [Discord](https://flipt.io/discord) or [Twitter](https://twitter.com/flipt_io).

Cheers and happy graphing 📈!


Grafana Dashboards


<BlogImage
  src="/auth-v1/auth.png"
  alt="Flipt Authentication"
  width={2400}
  height={1350}
>
  <span>Photo by [FLY:D](https://unsplash.com/@flyd2069) on Unsplash</span>
</BlogImage>

Last week we cut the [v1.15.0 release](https://github.com/flipt-io/flipt/releases/tag/v1.15.0) of Flipt. The primary feature of this release was authentication.
As of v1.15.0, Flipt has support for static client token-based authentication.
This is our first step to bringing authentication and authorization features to Flipt.

Currently, the way to obtain a client token is to enable the “token” authentication method in Flipt’s configuration.
Once enabled, a section of the API ([/auth/v1/method/token](https://www.flipt.io/docs/reference/authentication/create-token)) is mounted and available to create new access tokens.

Note that this endpoint is also protected when authentication is set to required, meaning prior authentication is needed to create tokens.

```yaml
authentication:
  methods:
    token:
      enabled: true
```

Check out our [token authentication configuration](https://www.flipt.io/docs/configuration#token) documentation for more specific details.

If no tokens exist in the backing store then one will be created the first time you enable the token authentication method.
This way, an initial token can be bootstrapped so that you don’t lock yourself out of Flipt (authentication can also be set to not required in this situation).
The bootstrapped token will be output in the logs:

```
…
2022-11-22T12:02:10Z    INFO    access token created    {"server": "grpc", "client_token": "nCq9QO-FeEAS91LV1LFD2qOpUASXl8eljlXokeorbyY="}
…
```

By default Flipt remains open, however, with a slight adjustment to your instance configuration, Flipt’s API can be secured.

```yaml
authentication:
  required: true
```

Attempts to perform API operations will require a client token to be presented via the Authorization header:

```sh
curl localhost:8080/api/v1/flags \
   -H 'Authorization: Bearer nCq9QO-FeEAS91LV1LFD2qOpUASXl8eljlXokeorbyY=' \
   -H 'Content-Type: application/json'
```

Putting this altogether we get a secured API with the ability for prior authenticated clients to create additional credentials:

```yaml
authentication:
  required: true
  methods:
    token:
      enabled: true
```

Head over to the new [Authentication](https://www.flipt.io/docs/authentication) section of our documentation to learn more about interacting with Flipt when authentication is required.

As mentioned, this is our first step to securing Flipt. Once authentication is enabled, the UI becomes unusable since we have not yet implemented the features required to authenticate browser sessions.

In the next release, we expect to add Google as the first OAuth login provider, followed by other authentication providers in subsequent releases.

Equally, we’re working on automated processes for managing expired client tokens.
With v1.15.0 it is currently on the operator to use Flipt’s APIs to delete outdated tokens, however, we plan for this mechanism to be built-in, automated, and configurable in the future.

On top of all this, we’re actively designing authorization features for Flipt, so that you can decide who can do what when interacting with the API and UI.

Join us in our [Discord](https://flipt.io/discord) to chat about your needs and to help influence the future of authentication and authorization with Flipt.


Authenticating Flipt


<BlogImage
  src="/openfeature-go/openfeature.png"
  alt="Flipt + OpenFeature"
  width={2400}
  height={1350}
/>

## What is OpenFeature?

[OpenFeature](https://openfeature.dev/) is a [CNCF Sandbox project](https://www.cncf.io/projects/openfeature/) that is developing an open standard for feature flag management.

> OpenFeature provides a unified API and SDK, and a developer-first, cloud-native implementation, with extensibility for open source and commercial offerings.

In short, OpenFeature provides a specification and implementation of a unified API to query and evaluate feature flags which abstract away the specific details of the backend responsible for storing and managing the flags themselves.

This allows the developers who are adding feature flags in their applications to use the OpenFeature SDK in their code and then register their feature flag management solution of choice as the 'provider', abstracting it away. The benefits of this are twofold:

1. It guards against 'vendor lock-in' which would require a large code refactor if a team were to decide to switch from one feature flag solution to another down the road.
1. It allows for common functionality to be shared and maintained by the community, which reduces the amount of duplication required as well as generally increases the quality and velocity of development. More on this later.

## SDKs and Providers

Before we get into more details of how OpenFeature works, it's important to define the key terminology and develop a shared model.

As mentioned above, OpenFeature has defined a [specification](https://docs.openfeature.dev/docs/specification/) that serves as a guide for how to think about OpenFeature from an end user perspective as well as how to implement a vendor specific library that OpenFeature can integrate with.

The specification defines '[providers](https://docs.openfeature.dev/docs/reference/concepts/provider)' as the adaptive layer that translates to and from feature flag solutions, such as Flipt and other open source and commercial solutions. It's up to the community or the feature flag vendors/projects themselves to build these providers that conform to the [OpenFeature Provider specification](https://docs.openfeature.dev/docs/specification/sections/providers).

<BlogImage
  src="/openfeature-go/openfeature-provider.png"
  alt="OpenFeature Provider"
  width={1654}
  height={340}
/>

An OpenFeature SDK is available for many popular programming languages such as **Go, JavaScript, .NET and Java**. The OpenFeature team just recently announced that these SDKs [have all reached a 1.0 version](https://docs.openfeature.dev/blog/hardening-and-1-0-sdks/), representing a certain maturity and resistance to breaking changes. They've also announced that more SDKs will be made available in the future for languages such as **PHP, Python, and Ruby**.

## Evaluation API

The [Evaluation API](https://docs.openfeature.dev/docs/specification/sections/flag-evaluation/) is the primary interface to access and interact with feature flags in the OpenFeature ecosystem. The OpenFeature Evaluation API allows you to query feature flags given a `flagKey`, default value, as well as an optional [evaluation context](https://docs.openfeature.dev/docs/specification/sections/evaluation-context). The evaluation context allows for the feature flag solution (provider) to make decisions based on configured rules, for example end-user targeting or percentage-based rollouts.

This maps nicely to how we think about [evaluation at Flipt](https://www.flipt.io/docs/concepts#evaluation), where the integrator can provide as much or as little information as they need to make a decision for the purpose of flag evaluation.

## Announcing The Flipt OpenFeature Provider for Go

Today we are happy to announce that we've **open sourced** the initial version of the [OpenFeature Go Provider for Flipt](https://github.com/flipt-io/flipt-openfeature-provider-go)! It supports both HTTP(s) and GRPC transports to connect to your Flipt instance and is highly configurable allowing you to bring your own HTTP or GRPC clients.

To ensure that we conform to the OpenFeature Provider specification, we even decided to pull in OpenFeature's own [integration test suite](https://github.com/open-feature/test-harness/tree/5153eac71dc47413da4326bdb49a94e573b99578) into the repository. This move actually helped us find several bugs in our implementation, while exercising it as an end-user would.

To see our provider in action, checkout the [example in our main repository](https://github.com/flipt-io/flipt/tree/main/examples/openfeature) which shows how to integrate OpenFeature + our provider into an example REST API project. Stay tuned for updated documentation, more examples, and future providers written in other languages. Also, if you are itching to write your own provider for Flipt in your language of choice, feel free to reach out in our [Discord](https://discord.com/invite/kRhEqG2TEZ), we'd love to help!

## Hooks and the Power of a Common API

Before wrapping up, I'd like to quickly go over one of what we think is the most powerful features of OpenFeature, the [Hooks API](https://docs.openfeature.dev/docs/specification/sections/hooks). Hooks allow you to use a common API for performing actions at different stages of the evaluation lifecycle. By implementing Hooks that use the OpenFeature data model, the community can take advantage of shared implementations, preventing the need for duplicating effort for each provider. Some of the use cases for hooks include things like adding metrics and telemetry, request retries and circuit breaking, performing validation on data, and performing client side caching.

The OpenFeature team has released their own [OpenTelemetry Hook for Go](https://github.com/open-feature/go-sdk-contrib/tree/main/hooks/open-telemetry), allowing any integrator of the Go SDK to get [OpenTelemetry](https://opentelemetry.io/) traces with the evaluated feature flag data for free. In our [example application](https://github.com/flipt-io/flipt/tree/main/examples/openfeature) for integrating the Flipt OpenFeature provider we wired up the OpenTelemetry hook with a few lines of code, resulting in some cool visualizations of our feature flag data:

<BlogImage
  src="/openfeature-go/jaeger-openfeature-otel-trace.jpg"
  alt="OpenFeature OpenTelemetry Trace"
  width={1582}
  height={399}
/>

<BlogImage
  src="/openfeature-go/jaeger-openfeature-otel.jpg"
  alt="OpenFeature OpenTelemetry Trace"
  width={1305}
  height={390}
/>

We are looking forward to seeing what the community comes up with by leveraging hooks to provide common functionality for all implementers of the OpenFeature SDKs!

## Committed to Helping Organizations Unlock the Power of Feature Flags

We love feature flags here at Flipt and think that we have just started to scratch the surface of what can be accomplished with their use in the software development lifecycle. With the introduction of the OpenFeature project, we hope that teams can begin to realize the full potential of feature flags in a way that is easy to implement and use. We're also very excited to see how the OpenFeature project evolves in regards to the CloudNative space, something we're also particularly interested in here at Flipt.

If you have any feedback on our provider, want us to prioritize specific language support, or just to learn more about how Flipt can help your organization get started with feature flags, please reach out either by joining our [Discord](https://discord.com/invite/kRhEqG2TEZ) or send us an email at [hello@flipt.io](mailto://hello@flipt.io).


Announcing the Flipt OpenFeature Provider for Go


<BlogImage
  src="/hi-george/geo.png"
  alt="Hello, George!"
  width={2400}
  height={1350}
>
  <span>
    Photo by [Lampos Aritonang](https://unsplash.com/@lamposaritonang) on
    Unsplash
  </span>
</BlogImage>

I'm excited to announce that [George MacRorie](https://www.linkedin.com/in/geokeaton/) has joined Flipt as our first Founding Software Engineer! George is a seasoned engineer with over 10 years of experience building and scaling software products. George and I first met at CodeShip, where we worked together for 3 years. George is based in the UK 🇬🇧 and will be key to developing and scaling Flipt as well as helping to build out our engineering team and remote first culture. George is an amazing engineer and a great person and friend, and I'm excited to have him on the team!

I asked George to write a bit about himself and his background, and here's what he had to say:

> My name is George. I’ve been bashing the internet pipes in exchange for money for over a decade. I cut my teeth as a junior engineer in the world of Java (and a lot of XML) in 2012. Before abandoning it all for my new love, the Go programming language, in 2014. Somewhere along the way I discovered the root of all my frustrations didn’t come from within, it was with my tools. Yeah, that's right, the tools were the problem. Thus, a developer tooling engineer was born.
>
> For me, It all started with CI/CD. I don’t think it is a coincidence that I embraced baldness around that time. Hair pulling was a known side effect of the available solutions back then. I joined CodeShip to build the alternative of our dreams. The CI/CD SaaS market was heating up at the time and CircleCI was already out in front, a position they’ve held well. In the end, our little ship didn’t quite cut the mustard. We didn’t get to fulfil our dreams of a CI/CD revolution.
>
> Yet, the best part about my time there was the people. We met many times in person (we were remote) all over Europe and North America. We shared ideas, laughs, and one hair-raising e-bike ride around the city of Lisbon. An experience which went down like Marmite. It was during this time that I met Mark. It was also during this time that Flipt came to be.
>
> When it all came to an end, Mark went off to continue the good developer tooling fight at GitHub. I and a few other ex-shippers figured we had solved CI/CD, so we should do databases next (InfluxDB). At Influx, I soon fell into my old habits of trying to solve developer tooling problems. They let me attempt to lead the Deployments team. A team which tackles many things’ platform engineering, despite the name.
>
> Feature flags at Influx are critical to how engineers develop in production. I wasn’t personally involved in the implementation of feature flags for Influx Cloud; it was another member of the old CodeShip crew who unlocked this capability. Unquestionably, they delivered on enabling the team to safely ship changes.
>
> Over time, our flagging libraries and frameworks accrued a little kludge. Some flags became entrenched and forgotten in code. Complexities in targeting logic grew. Local development and remote environments diverged in flag state configuration. Developers became suspicious if their flags were working as expected. One uncompleted large-scale refactor left the system fractured.
>
> Along the way, I got to see how the frustration played out, and I experienced it first-hand. I can’t say that has left me with all the answers. But, I would like to think it has left me with empathy. Something I intend to bring to Flipt.
>
> When Mark reached out to me and said, “want to come and build the future of feature flags?” I felt very ready for the challenge.

Welcome to the team, George! 🎉 🎉 🎉


Meet George MacRorie


<BlogImage
  src="/all-in/hello-world.png"
  alt="Hello World"
  width={1792}
  height={1024}
>
  <span>
    Art by [Alejandro Ortiz](https://unsplash.com/@alejandroortiz) on Unsplash
  </span>
</BlogImage>

## The Origins of Flipt

When I started [Flipt](https://github.com/flipt-io/flipt) almost 4 years ago, I started it to scratch my own itch. Throughout the previous 10 years of my career as a software engineer, I noticed that every company I'd worked at that used feature flags seemed to run into similar issues.

From the outside, the idea of feature flags seems simple enough. At their most basic, they are boolean values used to make decisions in your code that can change 'remotely' at runtime. Because of this perceived simplicity, most engineering organizations that I'd been a part of had decided to roll their own solution. This usually took the form of either an internal library developed in-house or an open-source library that was integrated into the main application and added to haphazardly over the years.

This approach works fine when feature flags are used sparingly and the team and application architecture are small enough. Once the organization and architecture grow, however, the simple, in-house solutions start to show their cracks. In most cases, the solution that works when there is a monolithic application no longer scales with the move to multiple services and programming languages. As the team begins to build out more complex features that require many different moving parts and dependencies, it becomes necessary for access to feature flags to be shared between these different services and teams.

At the time, I was aware of existing solutions that aimed to solve this same problem of distributed feature flags, however, they were all mostly SaaS-based. To me, this was not an ideal solution as I believed that most teams would want to avoid the lock-in, cost, security, and latency concerns associated with these solutions. Back in 2018, [gRPC](https://grpc.io/) was still a relatively new technology but it seemed like a promising solution for creating a reliable remote access layer that could be used to solve the above-mentioned problem of distributed access to feature flags in a multitude of programming languages.

Thus, Flipt was born, a stand-alone, self-hosted, feature flag service that allows developers to make use of feature flags in any programming language or framework while being focused on low latency and data residency from the start.

## Going All In

Over these past several years, myself and an amazing group of [contributors](https://github.com/flipt-io/flipt/graphs/contributors) have continued to make improvements to Flipt, adding features and support for new technologies and use cases. Over this same time, I've also talked with engineers, managers, and engineering leaders that use Flipt in production at their companies, helping provide real value to their end-users.

While this has been incredibly motivating and gratifying, it's also presented a bit of a challenge to keep up with all the new things I want to support and features to add while only being able to work on Flipt in my spare time. Thankfully, I've recently been given the opportunity to be able to work on Flipt full-time and also build out a small team to help make shipping software easier and less error-prone for developers everywhere.

The software industry has also come a long way over these past several years. More and more companies have fully embraced the CloudNative software development model, which has allowed us to build systems that are more resilient and performant than ever before. There's also been an explosion of new open source projects for developers to use as building blocks for their applications, which has made it much easier to get started with many common tasks such as testing, logging and monitoring. All of this has helped to make software development a more enjoyable, productive experience. However, there's still one major area where things haven't changed as much: shipping code safely. Despite all the advances we've seen in other areas, it remains just as difficult for developers to push their code changes into production while being assured that their changes will not cause errors.

This is the area where we want to focus our efforts with Flipt. We want to make it as easy and stress-free for developers to ship software as confidently as possible, regardless of their development environment, language, or platform.

We also want to continue to stay true to the project's core principles so that Flipt:

- Remains open-source and self-service.
- Stays flexible enough to support various modes of experimentation.
- Is seamless to configure and secure in modern cloud environments.
- Fits into your existing stack.
- Continues to be fast and consistent.

## Looking Forward

We're working on some really exciting features right now that we'll be releasing over the next few months, so keep an eye out for those!

I'm incredibly excited to take on this new challenge, embrace this opportunity, and build a team of like-minded individuals who are passionate about shipping code safely and securely.

**Takeaway**: We want to build the future of modern software deployment mechanisms, and we'd love to have you along for the ride. Flipt is not going anywhere and will remain open-source and community-driven. We are committed to making Flipt even better and would love your help! Please join us on our [GitHub](https://github.com/flipt-io/flipt) and in our [Discord](https://discord.com/invite/kRhEqG2TEZ) to learn more and help build the future of Flipt.

Thank you for your support, and we can't wait to see what you build with Flipt!
