Chapter 6: Data Integrity and Corruption Detection #

In the last chapter, WAL and Backend Snapshot Storage, you saw how etcd stores every Raft entry in a Write-Ahead Log (WAL) and how it applies those entries to a backend database. But how does etcd know that the backend database has not been corrupted? How can you prove that a snapshot backup is valid? What happens when one etcd member secretly has different data than the others?

This chapter is about data integrity and corruption detection.

Think of a library.

• The WAL is the librarian’s daily journal. Every time a book is moved, the librarian writes it down.
• The backend database is the shelf of books right now.
• The consistent index is a catalog label that says: “This shelf is up to date through journal entry 42.”

Now imagine the catalog says entry 42 is done, but the shelf does not actually contain the book from entry 42. That is corruption. The library looks fine, but a book is missing or wrong. Data integrity checks are like an auditor checking the catalog against the shelves, book by book.

By the end of this chapter, you will know how to verify an etcd snapshot, compare hashes across members, and understand why the v3.5 corruption bug taught the etcd team some hard lessons.

Our Central Use Case #

Suppose you are taking care of a small etcd cluster. One night, a server crashes. After restarting it, you start to worry:

• Is this member’s database the same as the other members?
• Is my offline snapshot backup usable?
• Would I know if one member silently lost a write?

In this chapter, we will walk through a simple health check:

1. Save a snapshot from the cluster.
2. Inspect the snapshot file with etcdutl.
3. Compare key-value hashes across live members.
4. Enable automatic corruption checks.

This is a great habit to learn, because corruption detection is what saves you from silent data loss.

Key Concepts #

1. Data Integrity #

Data integrity means that the data on disk is exactly what it should be. There are two important checks in etcd:

• The backend database must match the WAL at the consistent index.
• The logical key-value state must match across members.

If a member’s database is missing one write, the member still works. It can still serve reads. But it has silently diverged from the rest of the cluster. That is the most dangerous kind of bug.

2. Hash: A Fingerprint #

A hash is a short fingerprint computed from data. Give the same data to the same hash function, and you get the same fingerprint. If even one byte changes, the fingerprint is very likely to change.

etcd uses hashes in several places:

• snapshot status shows a hash of the snapshot database file.
• HashKV computes a hash of the keys and values, not the raw file.
• etcd-dump-db hash computes a hash of a backend database file.

When hashes match, you can be reasonably confident the data is identical.

3. Consistent Index: The Bookmark #

As you learned in the previous chapter, the backend database stores a consistent index in its meta bucket. This number says: “I have applied every WAL entry up to this index.”

On startup, etcd reads the consistent index and replays only the WAL entries after it. That is why the consistent index must be stored atomically with the data it describes. If the index is ahead of the data, etcd will think an entry was applied when it was not.

4. Logical Hash vs File Hash #

There is a difference between:

• File hash: a hash of the bytes in the database file. Useful for checking that two backup files are identical.
• Logical KV hash: a hash of the actual keys and values stored inside the database. Useful for checking that two cluster members contain the same logical data.

If you copy a snapshot file, the file hash should stay the same. If you compare two running members, use the logical KV hash.

5. Corrupt-Check Features #

etcd has experimental flags that ask members to compare their key-value hashes. If they disagree, etcd can raise a corruption alarm instead of silently serving incorrect data.

Those checks are important because a single-member cluster has no one to compare against. With multiple members, HashKV becomes your best friend.

Solving the Use Case Step by Step #

Let’s pretend we have a healthy snapshot called backup.db.

Step 1: Save a Snapshot #

From the live cluster, save a snapshot with etcdctl:

etcdctl snapshot save backup.db
# Output: "Snapshot saved at backup.db"

If you need a refresher, see etcdctl / etcdutl Command-Line Tools.

Step 2: Inspect the Snapshot File #

Now use etcdutl to inspect the file offline:

etcdutl --write-out=table snapshot status backup.db

Output example:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| cf1550fb |        3 |          3 | 25 kB      |
+----------+----------+------------+------------+

The HASH is a fingerprint of the snapshot file. If you copy this file to another machine, the hash should stay the same. The REVISION tells you how current the snapshot is.

Step 3: Compute a Logical Key-Value Hash #

A file hash is useful, but it does not tell you whether the contents are logically correct. Use etcdutl hashkv to hash the actual key-value pairs up to a revision:

etcdutl hashkv backup.db
# 35c86e9b, 214, 150

The output means:

• 35c86e9b is the key-value hash.
• 214 is the revision where the hash was computed.
• 150 is the compact revision.

If two snapshots have the same hash at the same hash revision, their key-value state is identical.

Step 4: Compare Live Members #

Now check the live cluster. etcdctl endpoint hashkv asks each member for its logical key-value hash:

etcdctl endpoint hashkv \
  --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379

Output example:

http://127.0.0.1:2379, 35c86e9b, 214, 150
http://127.0.0.1:22379, 35c86e9b, 214, 150
http://127.0.0.1:32379, 35c86e9b, 214, 150

All three members report the same hash, at the same revision. The cluster looks healthy.

If one member reports 35c86e9b and another reports deadbeef, you have a problem. One member has different key-value data than the others. You should investigate that member’s data directory with etcd-dump-db, and possibly restore it from a trusted snapshot.

Step 5: Inspect a Data Directory Directly #

If you have access to the data directory itself, you can compute a raw database file hash:

etcd-dump-db hash /tmp/etcd-data

Output example:

db path: /tmp/etcd-data/member/snap/db
Hash: 3700260467

This is useful when etcd is not running and you want to check whether a member’s database file matches another member’s file.

Step 6: Enable Automatic Corruption Checks #

You can also let etcd perform periodic hash checks automatically.

etcd \
  --experimental-initial-corrupt-check=true \
  --experimental-corrupt-check-time=10m

With these flags, etcd checks for corruption when it starts and then every 10 minutes. If a member reports a different hash, etcd can raise an alarm instead of silently continuing.

These flags are experimental, so read the release notes for your etcd version before relying on them.

What Happens Under the Hood? #

A Code-Light Walkthrough #

Let’s trace what happens when you run etcdutl hashkv backup.db.

etcdutl opens the bbolt database file in read-only mode. It looks inside the key bucket, which contains revisions of all key-value pairs. It then feeds the keys and values into a hash function up to the requested revision. Finally, it prints the hash and the revision.

Here is a simple sequence diagram:

When you run etcdctl endpoint hashkv, etcd sends a gRPC request to each member. The member computes the same kind of logical hash from its own backend database and returns it.

Why Atomicity Matters #

Remember the consistent index from the previous chapter? It must be updated in the same transaction as the data.

Here is a simplified version of that apply code:

// Simplified from server/storage/mvcc/kvstore.go
func (s *store) applyEntry(ent raftpb.Entry) error {
    req := pb.PutRequest{}
    req.Unmarshal(ent.Data)
    txn := s.backend.BatchTx()
    txn.UnsafePut(buckets.Key, req.Key, req.Value)
    txn.UnsafePut(buckets.Meta, consistentIndexKey, ent.Index)
    return txn.Commit()
}

The key-value write and the consistent index write are committed together. If the process crashes after Commit, both are saved. If it crashes before Commit, neither is saved. This “all or nothing” behavior is what protects etcd from WAL/database mismatches.

How a Hash Is Computed #

The real hash code is more complex, but the idea is simple:

// Simplified idea behind hashkv
func hashRevision(s *store, rev int64) (uint32, int64) {
    h := fnv.New32a()
    // For each key-value with revision <= rev:
    h.Write(v.Key)
    h.Write(v.Value)
    return h.Sum32(), rev
}

The exact functions live in etcd’s storage code, but you do not need to memorize them. What matters is that the hash covers the logical key-value state, not just the raw file.

The v3.5 Data Inconsistency Postmortem #

In 2022, the etcd team published a famous postmortem about a data corruption bug in etcd v3.5. You can read it here:

Documentation/postmortems/v3.5-data-inconsistency.md

What Went Wrong? #

A code refactor made the consistent index not be saved atomically with the data. In simplified terms:

1. etcd started applying a WAL entry.
2. Before the entry was fully applied, an in-memory consistent index value was updated.
3. A periodic commit triggered a hook that saved that in-memory value into the database.
4. Then the process crashed before the actual WAL entry was applied.

When the member restarted, it read a consistent index that was ahead of the data. It thought the WAL entry had already been applied, so it skipped it. The write was silently missing.

This is exactly the library catalog problem: the catalog said a book was on the shelf, but the book was not there.

What Did the Community Learn? #

The postmortem led to several important action items:

• Improve corruption detection so issues are found automatically.
• Add regression tests that reproduce historical data corruption bugs.
• Make the apply code easier to understand and validate.
• Build a stronger Robustness Testing Framework that deliberately crashes etcd and looks for inconsistencies.

For new contributors, the lesson is: storage code must be boring. A tiny change in transaction ordering can cause silent data loss across a distributed system.

Tools and Source Files to Explore #

If you want to see the real tools and code, start here:

• etcdutl/README.md — documents snapshot status, hashkv, and defrag.
• tools/etcd-dump-db/README.md — documents hash, list-bucket, and iterate-bucket.
• server/storage/mvcc/ — contains key-value storage and hashing logic.
• server/storage/backend/ — contains the bbolt backend wrapping.

For example, etcd-dump-db can show you the buckets inside a database:

etcd-dump-db list-bucket /tmp/etcd-data

Output:

alarm
auth
authRoles
authUsers
cluster
key
lease
members
members_removed
meta

The key bucket is where the actual key-value revisions live. The meta bucket stores metadata such as the consistent index.

Important Gotcha: Compare Same Revisions #

When comparing hashes across members, always make sure you are comparing at the same revision.

A slow member might legitimately be behind the leader. If it has not yet applied revision 214, it cannot produce the hash for revision 214. That is a lag problem, not necessarily corruption.

HashKV returns both the hash and the revision it used. If the revisions differ, you need to wait and check again.

Conclusion #

In this chapter, you learned:

• Data integrity means the backend database, WAL, and other cluster members all agree.
• Hashes are fingerprints for detecting corruption.
• etcdutl hashkv checks logical key-value data offline.
• etcdctl endpoint hashkv compares key-value hashes across live members.
• The consistent index must be stored atomically with the data it describes.
• The v3.5 postmortem showed what happens when that atomicity is broken.
• Automatic corrupt-check flags can help detect problems before they become silent data loss.

New contributors working on storage should always think about integrity. A member that does not fail loudly might fail silently, and silent failure is much scarier.

The next question is: can you trust the data that arrives over the network? After all, corruption detection only helps if the data itself comes from an authorized source.

Continue to Security and Trust Boundaries.

Generated by AI Codebase Knowledge Builder