Chapter 10: Robustness Testing Framework #

Welcome back! In the previous chapter, Multi-Module Go Repository, you learned how etcd organizes its code into separate Go modules. That keeps dependencies clean and makes the project easier to maintain.

But a clean codebase is not enough. How do we know etcd’s distributed promises really hold? How do we prove that a successful write is still there after a crash, or that watches never travel back in time? That is exactly what the Robustness Testing Framework is for.

Think of instant replay in sports:

• Every play is recorded on video.
• Referees compare the replay to the rulebook.
• If a player breaks the rules, the referees catch it.

The robustness framework does the same thing for etcd:

• It records every client operation as a play.
• The rulebook is a formal consistency model.
• The referees are validators that compare the recorded history to the rulebook.

If the history does not match the model, the test fails and saves a detailed report.

Our Central Use Case #

Imagine you are fixing a bug in etcd’s storage layer. Your change is small, but storage bugs can be dangerous. Even a one-line change might make etcd lose acknowledged writes after a crash.

How do you prove your change is safe?

You run a regression test:

make test-robustness-issue14370

This command starts a real etcd cluster, generates real client traffic, crashes a member, and then validates that every recorded operation still makes sense. If your change broke etcd’s guarantees, the test fails.

By the end of this chapter, you will know how that test works and how to analyze its report.

Key Concepts #

Let’s break the framework into small, friendly pieces.

1. Distributed Guarantees #

etcd promises certain behaviors even when things fail. Two important ones are:

• KV API guarantees: Operations like Put and Get are strictly serializable. Once a write succeeds, later reads should see it, and the revision number should never go backward.
• Watch API guarantees: Watches are eventually consistent. Events may be delayed, but they should never be lost, duplicated, or reordered in a way that breaks etcd’s watch rules.

These guarantees are like the rules of a sport. The robustness framework makes sure etcd follows them.

2. Failure Injection #

It is not enough to test the happy path. Real etcd clusters experience:

• Process crashes.
• Network partitions between members.
• Disk latency or slow disks.
• Compactions and defragmentations happening at the wrong moment.

The framework injects failures on purpose. It may kill a member with SIGKILL, cut a network link, or pause a process. This is like a referee secretly turning off the stadium lights during a game. If the players still follow the rules, they pass.

3. Operation History #

The framework records every client operation:

• What request was sent.
• Which member received it.
• When it started.
• When it finished.
• What response was returned.

This recorded history is the “instant replay” video.

4. Formal Consistency Models #

A formal consistency model is a simplified description of how etcd should behave. It asks questions like:

• Can this successful Put be placed inside the history without breaking the order?
• Can this failed read return this revision?
• Can this watch deliver this event here?

The validator checks whether the recorded history could possibly exist under the model. If not, the framework reports a bug.

5. Regression Commands #

Some historical bugs are so important that etcd includes reproduction commands for them. For example:

• make test-robustness-issue14370 reproduces the “single node cluster loses a write after crash” bug.
• make test-robustness-issue15271 reproduces the “watch traveling back in time after network partition” bug.

These commands make sure the framework still catches old bugs after new changes.

6. Track Record #

The robustness README contains a table of real bugs discovered by the framework. Here are a few examples:

These are not made-up tests. These are real bugs that the framework found.

Solving the Use Case: Run a Regression Test #

Let’s go through the process of running a robustness regression test.

Step 1: Build etcd with failpoints #

Robustness tests need to inject failures at precise moments. etcd uses gofail markers for that.

Run these commands from the etcd repository root:

make gofail-enable
make build
make gofail-disable

gofail-enable creates special failure injection points in the code. Then build creates a binary with those points. gofail-disable cleans up afterward.

If you don’t build with failpoints, the tests cannot crash etcd at the most interesting moments.

Step 2: Run a regression test #

Now run the reproduction command for issue #14370:

make test-robustness-issue14370

This test:

1. Starts a one-node etcd cluster.
2. Sends multiple clients issuing Put, Get, and other operations.
3. Kills the etcd member while traffic is running.
4. Restarts the member or lets the cluster recover.
5. Validates the recorded operation history.

Because the bug is timing-dependent, the test may pass many times before it fails. To increase the chances, use:

GO_TEST_FLAGS='--count=100 --failfast' make test-robustness

--count=100 runs the test 100 times. --failfast stops at the first failure.

Step 3: Read the verdict #

If the test finds a problem, you will see logs like this:

Validating linearizable operations {"timeout": "5m0s"}
Linearization illegal
Saving robustness test report {"path": "/tmp/TestRobustnessRegression_Issue14370/..."}
Saving visualization {"path": "/tmp/TestRobustnessRegression_Issue14370/.../history.html"}

The important line is Linearization illegal. That means the validator found an operation history that cannot happen under etcd’s consistency guarantees.

If the test passes, you will see something like:

Linearization success

Step 4: Analyze the report #

The report directory contains everything you need to debug:

/tmp/TestRobustnessRegression_Issue14370/
├── server-test-0/
│   ├── member/wal/
│   └── member/snap/db
├── client-1/
│   ├── operations.json
│   └── watch.json
└── history.html

• history.html is a visual timeline of operations.
• client-*/operations.json is the recorded KV history.
• client-*/watch.json is the recorded watch history.
• server-*/ contains the etcd data directories, which you can inspect with tools from WAL and Backend Snapshot Storage and Data Integrity and Corruption Detection.

Open history.html in your browser. Click “jump to first error”. You will see the exact operations that violate the model.

What Happens Under the Hood? #

Now let’s lift the hood and see how the machinery works.

Code-Light Walkthrough #

Here is a simple sequence diagram for a robustness test:

The test never just checks “did the cluster stay up?” It checks “was the history legal under the model?”

Deeper Look: A Simplified Test #

The real robustness tests live under tests/robustness/. The README in that directory is the best starting point.

Here is a very simplified mental model of how a regression test is written:

// tests/robustness/main_test.go (very simplified idea)
cluster := StartEtcdCluster(t, Config{Size: 1})
recorder := NewRecorder()

go RunTraffic(cluster, recorder) // Put/Get/Watch in background
InjectCrash(cluster)             // kill a member at a bad time

Validate(t, recorder.History(), cluster)

• StartEtcdCluster starts real etcd processes.
• NewRecorder records all client operations.
• RunTraffic sends realistic mixed traffic.
• InjectCrash crashes a member.
• Validate checks the history against consistency models.

A Simplified Validator #

The actual validator is very complex, but you can imagine it like this:

// Very simplified validator
func ValidateLinearizable(history []Operation) error {
    for _, op := range history {
        if op.Revision < op.PrevRevision {
            return fmt.Errorf("revision went backward")
        }
    }
    return nil
}

The real validator uses a formal linearizability checker. It tries to find a total order of operations that matches both the responses and the real-time order of requests.

The validated guarantees include:

• Successful writes appear exactly once and never disappear.
• Revisions never decrease.
• Reads return data from a consistent point in time.
• Watch events are ordered and never replayed incorrectly.

Why Failpoints Matter #

Failpoints are special markers in the code that let tests inject failures at exactly the right spot.

For example, before an entry is applied to the backend database, etcd can pause, crash, or sleep. That lets tests explore the tiny window where bugs hide.

This is important because of real-world incidents like the v3.5 data inconsistency issue described in WAL and Backend Snapshot Storage. A fix for that bug is only trustworthy if the framework can reproduce the historical failure first.

You can read the full postmortem in:

• Documentation/postmortems/v3.5-data-inconsistency.md

That document is an excellent example of why robustness tests exist.

Robustness vs Antithesis Tests #

etcd also works with Antithesis, a platform for deterministic simulation testing. Antithesis runs the same robustness tests inside a special environment that explores edge cases and race conditions.

You can think of Antithesis as a super-powered version of instant replay:

• The game is played over and over in a simulated universe.
• The referee can pause time, reorder events, and inject bizarre failures.
• If any play breaks the rulebook, the test records it.

The code lives in tests/antithesis/. It uses Docker Compose to start a 3-node etcd cluster plus a client container, then validates the collected history.

The Old Local Tester #

Before the full robustness suite, etcd had a smaller tool called tools/local-tester. It used goreman and unreliable network bridges to inject failures.

The README says:

etcd-local-tester is now deprecated in favor of our much more comprehensive robustness testing suite.

So if you see tools/local-tester/ in the repository, treat it as an old version of the idea. The modern framework is tests/robustness/.

Maintaining Historical Reproducibility #

The robustness framework is not only about finding new bugs. It is also about making sure old bugs can still be reproduced.

When etcd maintainers make big changes to the test framework, they follow a simple rule:

1. Before the change, run the regression commands for known bugs.
2. After the change, run them again.
3. If a regression command no longer catches its bug, the change is incomplete.

The README tracks this with the “Last reproduction commit” column. This tells future contributors which version of the framework was last proven to reproduce each bug.

This is why commands like make test-robustness-issue14370 exist. They are not just tests. They are promises that the framework remembers the past.

Running the Full Robustness Suite #

If you want to run all robustness tests locally, use:

make test-robustness

This runs many scenarios with different cluster sizes, traffic patterns, and failures.

You can adjust the run with environment variables:

GO_TEST_FLAGS='--count=100 --failfast' make test-robustness

There is also a coverage directory, tests/robustness/coverage/, that analyzes how Kubernetes uses etcd. This helps the etcd team ensure robustness tests cover the real Kubernetes–etcd contract.

Conclusion #

In this final chapter, you learned that:

• The robustness framework starts real etcd clusters and injects real failures.
• It records every client operation and compares the history to formal consistency models.
• It is like instant replay in sports: every play is recorded, and referees compare it to the rulebook.
• Regression commands like make test-robustness-issue14370 keep historical bugs reproducible.
• The framework has found many real bugs, including lost writes and watches traveling back in time.
• Failure injection uses gofail markers to crash at the most interesting moments.
• Antithesis extends robustness testing with deterministic simulation.
• Older tools like local-tester have been replaced by this more comprehensive suite.

The robustness framework is etcd’s safety net. It cannot prove that etcd is perfect, but it makes sure that when something breaks, we have a recording, a rulebook, and a referee to diagnose exactly what went wrong.

Thank you for reading this tutorial! You started with a simple clientv3 Put and traveled all the way through gRPC, Raft, WAL, security, feature gates, Go modules, and now robustness testing. You now have a strong foundation for exploring etcd’s code and contributing safely.

Generated by AI Codebase Knowledge Builder