Chapter 8: Feature Gates and Release Stages #

Welcome back! In the previous chapter, Security and Trust Boundaries, we learned how etcd protects the two main “gates” into a cluster: port 2379 for clients and port 2380 for peers. mTLS makes sure only trusted actors can get inside.

But there is another kind of gate inside etcd itself: a feature gate. It controls which new features are visible, active, and trustworthy. Feature gates are how etcd manages code that is still experimental while keeping stable users safe.

Imagine a theme park adding a brand-new roller coaster.

• The ride is built in a hidden area: Alpha. Only special guests can ride, and the park can remove it at any time.
• After months of testing, the park opens it to everyone: Beta. It is on the regular map, but it may still need fine-tuning.
• Finally, the ride becomes permanent: GA. It is always open, and you no longer need a special pass to find it.

This chapter shows you how etcd uses that same idea for software features.

Our Central Use Case #

You are a new contributor to etcd. You want to add a feature called FastLeaseKeepAlive that makes lease renewals faster. You are excited, but you are also new. How do you share your idea without accidentally breaking everyone who runs etcd?

You need a plan that:

1. Keeps your unfinished code hidden by default.
2. Tells users how much they can trust it.
3. Gives the project a clear path to move it from experimental to stable.
4. Allows the project to remove or replace it later if needed.

That plan is the feature gate and release stage system.

By the end of this chapter, you will understand how to add a feature, graduate it from Alpha to Beta to GA, and deprecate it when it is no longer useful.

Key Concepts #

Let’s break the system into small pieces.

1. What Is a Feature Gate? #

A feature gate is a switch in the etcd server code. It decides whether a piece of code runs or not.

For example, imagine the fast lease renewal path is only for brave users:

if cfg.ServerFeatureGate.Enabled(features.FastLeaseKeepAlive) {
    // use the new fast path
} else {
    // use the old, well-known path
}

If the gate is on, etcd uses the new code. If the gate is off, etcd uses the old code.

The gate is a safety switch. It lets developers ship code that exists in the binary but is not active for everyone.

2. Release Stages: Alpha, Beta, GA #

Every new feature starts with a release stage. The stage tells users how stable the feature is.

This is the staged rollout idea:

• Alpha is a pilot program.
• Beta is a standard offering.
• GA is permanent infrastructure.

3. Alpha: The Pilot Program #

Alpha features are off by default. Users must explicitly enable them, often with a flag like --feature-gates=FastLeaseKeepAlive=true.

Alpha features:

• Might be buggy.
• Are not recommended for production.
• Can be removed in the next release without warning.
• Do not need to follow the full deprecation process.

Think of Alpha as a “hidden menu” item at a restaurant. Only customers who ask for it can try it. If the chef decides it is not working, it disappears from the secret menu with no apology.

4. Beta: The Standard Offering #

When a feature proves useful, it can graduate to Beta.

Beta features:

• Are enabled by default.
• Are supported in etcd releases.
• Can no longer be removed without following the deprecation policy.

Users do not need to turn Beta features on. They are part of the normal etcd experience. However, they may still change as users find issues.

Think of Beta as a dish added to the regular menu. The restaurant now supports it, but the recipe can still be improved.

5. GA: Permanent Infrastructure #

When a feature is fully stable, it graduates to GA.

GA features:

• Are always enabled.
• Cannot be disabled.
• No longer need a feature gate.
• Must follow the deprecation policy before removal.

Think of GA as the restaurant’s signature dish. It is always on the menu, and there is no “off switch” for it.

6. KEP: The Proposal Process #

etcd uses a process inspired by Kubernetes called KEP, which stands for Kubernetes Enhancement Proposal.

Wait, etcd is not Kubernetes. Why use Kubernetes’ process?

Because the Kubernetes community built a very good system for proposing, tracking, and graduating large features. etcd borrows that process in a simplified form.

A KEP-style proposal should:

• Explain the problem clearly.
• List the work items.
• State graduation criteria for each stage.
• Stay open until the feature graduates or is dropped.

Two maintainers must approve the KEP and the code changes.

Think of a KEP as a business plan before opening a new restaurant location. You do not just start cooking; you write down what you will serve, why people need it, and how you will know it is successful.

7. Graduation Criteria: Proving a Feature Is Ready #

A feature should not stay Alpha forever. It should move to the next stage once it meets the goals in its KEP.

Before a feature can graduate, contributors need to provide:

• Unit tests.
• Integration tests where possible.
• E2E (end-to-end) tests for realistic coverage.
• Logs that help with debugging.
• Metrics and benchmarks if needed.
• A CHANGELOG entry.

Also, a feature should stay in one stage for at least one release before being promoted. Patch releases are not used for graduation.

8. Deprecation: Saying Goodbye #

Not every feature survives. When a feature needs to go away, etcd has a careful process.

• Alpha features can be removed immediately, without a long goodbye.
• Beta and GA features must be deprecated over two releases.

Why two releases? Because users may be relying on the feature. They need time to see the warning, change their configuration, and migrate away.

The deprecation process looks like this:

1. Announce the deprecation in release notes and feature gate docs.
2. In the next release, change the gate to Deprecated. If someone uses it, warn them.
3. In the release after that, lock the gate to its default and clean up the code.

This is like closing a popular restaurant. The owner does not just vanish overnight. They put a sign on the door, then host a final week, then lock the doors.

Solving the Use Case: Adding FastLeaseKeepAlive #

Let’s walk through the full life of our feature, step by step.

Step 1: Write a KEP-Style Proposal #

Your first step is not code. It is a document.

Open an issue and a PR in the KEP area with:

• A clear explanation: “Lease renewals take too long because we wait for the applied index.”
• Work items: “Add feature gate, add fast path, add tests.”
• Graduation criteria: “Feature graduates to Beta after benchmark shows no correctness regressions.”

Two maintainers review it. Once they approve, you can start coding.

Step 2: Add an Alpha Feature Gate #

Now you add the feature gate in the etcd code.

A simplified version looks like this:

// server/features/etcd_features.go
var FastLeaseKeepAlive = featuregate.Feature("FastLeaseKeepAlive")

func init() {
    FeatureGate.Add(featuregate.FeatureSpec{
        Default:    false,
        PreRelease: featuregate.Alpha,
    })
}

This code creates a feature named FastLeaseKeepAlive.

Because Default is false, the feature is off by default. Because PreRelease is Alpha, it is clearly marked as experimental.

Step 3: Gate the Code #

Next, you need to protect the new code path. You do not want it to run for everyone.

if cfg.ServerFeatureGate.Enabled(features.FastLeaseKeepAlive) {
    // fast path: skip waiting for applied index
    return renewLeaseFast()
}
// old path: wait for applied index
return renewLeaseOld()

Before this code runs, etcd checks the feature gate. If the feature is disabled, etcd uses the old path. If a user enables it, etcd uses the new path.

This is like having two routes to the same destination. One is the well-known highway. The other is a shortcut that is still being tested.

Step 4: Add Tests #

Every feature needs tests. You add:

• Unit tests for the fast path.
• Integration tests with a real etcd server.
• E2E tests that verify the feature works from the client side.

You also add a CHANGELOG entry for the new Alpha feature.

Step 5: Try It With the Feature Gate #

Now a brave user can try the Alpha feature by starting etcd with the feature gate turned on:

etcd --feature-gates=FastLeaseKeepAlive=true

etcd starts normally. The new feature is registered and reported in the server’s metrics.

You might see something like this when querying the metrics endpoint:

curl -s http://127.0.0.1:2379/metrics | grep etcd_server_feature_gates

etcd_server_feature_gates_enabled{name="FastLeaseKeepAlive"} 1

The output is simplified, but the idea is real: etcd tells operators which feature gates are enabled. This is useful for debugging and for knowing which path a server is using.

If the feature gate is off, the metric shows 0 instead of 1.

Step 6: Graduate to Beta #

After one release, users report that the feature works well. The KEP graduation criteria are met. Now you open a PR to change the stage to Beta:

// server/features/etcd_features.go
FeatureGate.Add(featuregate.FeatureSpec{
    Default:    true,
    PreRelease: featuregate.Beta,
})

Now the feature is on by default. Users no longer need to set the flag to try it. The feature is supported, but it can still be improved.

Step 7: Graduate to GA #

Later, the feature becomes fully stable. It graduates to GA.

For a GA feature, the gate is no longer needed because the feature is always on. In code, it becomes locked to its default:

// server/features/etcd_features.go
FeatureGate.Add(featuregate.FeatureSpec{
    Default:         true,
    PreRelease:      featuregate.GA,
    LockedToDefault: true,
})

From this point on, users cannot disable it. The feature is permanent infrastructure.

Step 8: Deprecate It (If Needed) #

Suppose a better feature replaces FastLeaseKeepAlive in the future. Now you have to say goodbye carefully.

In release N:

FeatureGate.Add(featuregate.FeatureSpec{
    Default:         false,
    PreRelease:      featuregate.Deprecated,
    LockedToDefault: false,
})

When someone tries to use the deprecated gate, etcd prints a warning.

In release N+1:

FeatureGate.Add(featuregate.FeatureSpec{
    Default:         false,
    PreRelease:      featuregate.Deprecated,
    LockedToDefault: true,
})

Now the gate is locked. The old code can be cleaned up safely.

What Happens Under the Hood? #

Now that you have seen the feature from the outside, let’s look at the machinery inside etcd.

Code-Light Walkthrough #

When etcd starts, it does something like this:

1. It creates a FeatureGate object.
2. Every feature calls FeatureGate.Add(...) to register itself.
3. etcd parses the --feature-gates flag.
4. For each registered feature, the gate checks the default stage and the flag.
5. Later, when a request comes in, the code calls Enabled(...).
6. If the gate returns true, the new code runs. If false, the old code runs.

Here is a simple diagram:

The FeatureGate is the middleman between the developer’s code and the user’s choice.

Deeper Dive: Where the Code Lives #

Feature gate code lives in the etcd server module. A simplified version of registration looks like this:

// server/features/etcd_features.go
package features

var FastLeaseKeepAlive = featuregate.Feature("FastLeaseKeepAlive")

func init() {
    FeatureGate.Add(featuregate.FeatureSpec{
        Default:    false,
        PreRelease: featuregate.Alpha,
    })
}

When the etcd server configuration is built, ServerFeatureGate is checked wherever the feature is used:

// server/etcdserver/lease_keepalive.go (simplified)
if cfg.ServerFeatureGate.Enabled(features.FastLeaseKeepAlive) {
    return s.fastKeepAlive()
}
return s.slowKeepAlive()

The important thing is that all code changes related to the feature must be gated. If a new flag, config field, or code path is added, it should be behind the gate.

This is how etcd hides incomplete code safely. The code exists in the binary, but the gate prevents it from running unless the user explicitly or implicitly enables it.

How the Gate Makes Decisions #

A feature gate is basically a map from feature name to a boolean value.

A very simplified mental model:

// mental model, not real etcd code
func (g *featureGate) Enabled(name string) bool {
    spec := g.features[name]
    value := spec.Default
    if override, ok := g.overrides[name]; ok {
        value = override
    }
    return value
}

If the user passes FastLeaseKeepAlive=true, the override is true. If no override is given, the default from the stage is used.

For Beta features, the default is true. For Alpha features, the default is false. For GA features, the value is locked to true and cannot be changed.

Metrics For Feature Gates #

Operators need to know which features are active. etcd exposes feature gate state as a Prometheus metric.

A simplified metric looks like:

etcd_server_feature_gates_enabled{name="FastLeaseKeepAlive"} 0

If the value is 1, the feature is enabled. If 0, it is disabled.

This is very useful when comparing two etcd servers that behave differently. One server might have an Alpha feature on; the other might not.

The CHANGELOG Connection #

Every feature change should be visible in the changelog. The etcd changelog files live under CHANGELOG/.

For example, when FastLeaseKeepAlive is added, the changelog entry might appear in CHANGELOG/CHANGELOG-3.7.md:

### etcd server

- Add FastLeaseKeepAlive feature to enable faster lease renewal...

Changelogs help operators know exactly what changed in a new release. Without them, nobody would know that a new Alpha feature exists.

Why This System Matters #

Feature gates and release stages are not just bureaucracy. They solve a real trust problem.

Imagine if every new feature were enabled by default immediately. One buggy feature could corrupt data, break consensus, or cause a cluster to crash. Users would lose trust in etcd.

Instead, etcd uses a staged rollout:

• Alpha hides the feature from normal users.
• Beta lets everyone try it, but the project still supports it.
• GA makes it a permanent promise.

This gives contributors confidence to experiment and gives users confidence to upgrade.

It also tells you, as a reader of etcd source, exactly how much you should trust a new capability. If a feature gate is Alpha, treat it like a prototype. If it is Beta, treat it like a tested product. If it is GA, treat it like infrastructure.

Conclusion #

In this chapter, you learned:

• A feature gate is a switch that turns code on or off.
• Features start in Alpha: off by default, experimental, and safe to remove.
• Features can graduate to Beta: on by default and supported.
• Features can finally reach GA: always on, and the gate disappears.
• New features should start with a KEP-style proposal and clear graduation criteria.
• Code changes must be gated with Enabled(...) checks.
• Features need unit tests, integration tests, e2e tests, metrics, and changelog entries.
• Deprecating a Beta or GA feature takes two releases.
• Alpha features can be removed at any time.

The feature gate system is how etcd says: “Here is something new. Try it carefully. When it proves itself, we will make it permanent.”

Now that you know how features are organized across releases, let’s look at how the etcd codebase itself is organized as a multi-module Go repository. Where do these feature gate files actually live?

Continue to Multi-Module Go Repository.

Generated by AI Codebase Knowledge Builder