Chapter 5: WAL and Backend Snapshot Storage #

In the previous chapter, Cluster Membership and Peer Communication, you learned how etcd members join a cluster and talk to each other on port 2380. But what happens when an etcd member crashes and restarts? How does it remember all of the keys and values you wrote?

The answer is etcd’s storage layer. It is made of two important pieces that work together:

• A Write-Ahead Log (WAL) — an append-only diary of every change.
• A backend database snapshot — a bbolt file that contains the state at a particular point in time.

Think of it like writing a book:

• The WAL is your daily journal. Every time you change a sentence, you write it down in the journal.
• The backend database is the latest clean copy of the book. It may not have your most recent edits, but it is close.
• If your computer crashes, you open the latest clean copy and then re-apply the journal entries after that point.

This chapter explains how that works, what the files look like, and why one tiny atomicity detail is extremely important.

Motivation: Remembering Everything After a Crash #

etcd is designed to be reliable. That means it must survive crashes without losing acknowledged writes. If etcd only kept data in memory, a restart would lose everything. If etcd only kept a current-state database, it might not know exactly how far the database is up to date.

The solution is to keep both:

1. An append-only WAL of all Raft log entries.
2. A backend database that stores the current state as of a specific “consistent index”.

The consistent index is the link between the two. It tells etcd: “The backend database already contains every log entry up to this point.”

On restart, etcd:

1. Opens the backend database.
2. Reads the consistent index from it.
3. Reads WAL entries after that index.
4. Replays those entries into the backend database.

This gives etcd both speed and safety. The database gives it a head start, and the WAL makes sure nothing important is lost.

Our Central Use Case #

Let’s see this in action.

Start an etcd member with a custom data directory:

etcd --data-dir /tmp/etcd-data

Now write a key twice:

etcdctl put app/config v1
etcdctl put app/config v2

The value is now "v2". Pretend the machine crashes. You can simulate this by killing the etcd process:

kill -9 12345

Then start etcd again with the same data directory:

etcd --data-dir /tmp/etcd-data

Read the key:

etcdctl get app/config
# output:
# app/config
# v2

etcd recovered the exact state. No special command was needed. The WAL and backend database did the work automatically.

Key Concepts #

1. The Write-Ahead Log (WAL) #

The WAL is an append-only log of Raft entries. Every time etcd processes a change, it first writes a Raft entry to the WAL. The entry contains things like:

• the term and index of the entry,
• the type of entry,
• the serialized request data, for example put:<key:"app/config" value:"v2">.

Because entries are append-only, writing them is fast. There is no random seeking, and no overwriting old data.

You can find WAL files in the data directory under member/wal/. The file names look like:

/tmp/etcd-data/
└── member/
    ├── wal/
    │   └── 0000000000000000-0000000000000000.wal
    └── snap/
        └── db

The member/snap/db file is the backend database.

2. The Backend Database (bbolt) #

The backend database is a bbolt file. bbolt is a persistent B+ tree key-value store. It stores the current etcd state in buckets. Some bucket names you will see are:

• key — the actual key-value data.
• lease — lease metadata.
• auth — authentication data.
• meta — metadata, including the consistent index.
• members — cluster member information.

The backend database is not just a WAL replay log. It is a real database file that etcd updates as it applies committed entries.

It is also the thing that etcdctl snapshot save copies. When you save a snapshot, you are saving a consistent copy of this backend database.

3. The Consistent Index #

Every Raft log entry has an index. This is just a number that increases by one for each new entry.

The backend database stores a special value called the consistent index. It says: “This database is current up to WAL entry number N.”

On startup, etcd reads the consistent index from the database. Then it replays only WAL entries after N. This avoids replaying entries that are already reflected in the database.

A good analogy is a bookmark in a long book. The bookmark says “I have read through page 100.” If the book changes, you do not re-read from page 1. You start at page 101.

4. Atomicity: The Most Important Rule #

The consistent index must be updated in the same transaction as the data it describes.

For example, when etcd applies WAL entry 32, it does two things:

• It writes the change from entry 32 into the key bucket.
• It updates the consistent index in the meta bucket to 32.

These two writes must happen together. If the value is written but the index is not updated, etcd will replay the same entry again after a crash. If the index is updated but the value is not written, etcd will skip the entry after a crash, and that change is lost forever.

You can think of this like updating a spreadsheet and a summary row. If you update the summary before the cells, your summary lies. If the summary says a row is finished when it is not, you will not go back and fix it.

What Happens When etcd Writes a Value? #

When you run etcdctl put, the request goes through Raft, as described in Raft Consensus Engine. Once Raft commits the entry, the etcd server applies it to the backend database.

Here is a simplified picture of that flow:

In this diagram:

• C is the client, perhaps etcdctl.
• E is the etcd server.
• W is the WAL file.
• B is the bbolt backend database.

The WAL entries are saved before etcd answers the client. That is why a successful write is still there after a crash.

What Happens When etcd Restarts? #

When etcd starts, it does the opposite movement:

1. Open the backend database.
2. Read the consistent index.
3. Find WAL entries after that index.
4. Replay them into the backend database.
5. Become ready for clients.

Here is a simplified diagram:

If the consistent index is 31, etcd replays entry 32 and any later entries. If the database is already fully up to date, there are no entries to replay and startup is very fast.

Inspecting WAL and Backend Files #

Sometimes you want to look inside these files. etcd has two small tools for that:

• etcd-dump-logs — reads WAL files and prints entries.
• etcd-dump-db — reads backend database files and prints buckets or records.

You can install them from the etcd source tree:

go install -v ./tools/etcd-dump-logs
go install -v ./tools/etcd-dump-db

Inspect the WAL #

This command dumps the WAL from a data directory:

etcd-dump-logs /tmp/etcd-data

The output is simplified in this example, but it looks roughly like:

WAL metadata:
nodeID=1 clusterID=... term=4 commitIndex=34 vote=0
WAL entries:
lastIndex=34
term  index  type  data
  3    31   norm  put:<key:"app/config" value:"v1">
  3    32   norm  put:<key:"app/config" value:"v2">

You can see the exact historical log entries that led to the current state.

You can also filter by entry type:

etcd-dump-logs -entry-type Normal /tmp/etcd-data

Inspect the Backend Database #

To list buckets in the database:

etcd-dump-db list-bucket /tmp/etcd-data

Typical output:

alarm
auth
authRoles
authUsers
cluster
key
lease
members
members_removed
meta

The key bucket contains the key-value data. The meta bucket contains metadata such as the consistent index.

You can also compute the hash of a database file:

etcd-dump-db hash /tmp/etcd-data
# db path: /tmp/etcd-data/member/snap/db
# Hash: 3700260467

Hashes are useful for comparing two databases. That leads us to the Data Integrity and Corruption Detection chapter.

Saving and Restoring a Backend Snapshot #

The backend database is also the basis for backup and disaster recovery. As you saw in etcdctl / etcdutl Command-Line Tools, you can save a snapshot and restore it:

etcdctl snapshot save backup.db
etcdutl snapshot status backup.db
etcdutl snapshot restore backup.db \
  --name member1 \
  --initial-cluster member1=http://127.0.0.1:2380

The snapshot file is a consistent copy of the backend database. If a whole cluster is lost, you can restore the snapshot into a new cluster.

Under the Hood: Code-Light Walkthrough #

Now let’s peek into the actual code, but keep it simple.

Saving a WAL entry #

The WAL code lives in server/storage/wal/wal.go. When etcd needs to persist a Raft entry, it appends it and then syncs the file to disk.

Here is a simplified version:

// server/storage/wal/wal.go (simplified)
func (w *WAL) Save(st raftpb.HardState, ents []raftpb.Entry) error {
	for _, e := range ents {
		data := pbutil.MustMarshal(&e)
		w.encoder.encode(&data)
	}
	return w.fp.Sync() // fsync to disk
}

Sync() is what makes the write durable. Without it, the entry might remain in memory and be lost during a power failure.

Applying an entry to the backend #

The backend code lives in server/storage/backend/backend.go, but the apply logic is in server/storage/mvcc/kvstore.go.

When etcd applies a committed WAL entry, it writes both the key-value change and the consistent index in one bbolt transaction:

// server/storage/mvcc/kvstore.go (simplified)
func (s *store) ApplyEntry(ent raftpb.Entry) error {
	req := pb.PutRequest{}
	req.Unmarshal(ent.Data)
	txn := s.backend.BatchTx()
	txn.UnsafePut(keyBucket, req.Key, req.Value)
	txn.UnsafePut(metaBucket, consistentIndexKey, ent.Index)
	return txn.Commit()
}

The Commit() call is the critical part. If the key-value write and the consistent index write are in the same transaction, they succeed or fail together. That atomicity is what protects etcd from data inconsistency.

Replaying the WAL on restart #

The recovery code in server/etcdserver/storage.go reads the consistent index and replays newer entries:

// server/etcdserver/storage.go (simplified)
func (s *EtcdServer) recoverFromDisk() error {
	ci := s.backend.ReadConsistentIndex()
	ents := wal.ReadEntriesAfter(ci)
	for _, ent := range ents {
		s.applyEntry(ent)
	}
	return nil
}

Again, this is simplified, but that is the whole idea.

The v3.5 Data Inconsistency Postmortem #

You might be wondering: “Is atomicity really that easy to break?”

Yes. In etcd v3.5, a code refactor caused the consistent index to not be saved atomically with the data. Under the right crash conditions, a member’s consistent index could be ahead of its actual data. When the member restarted, it thought some entries were already applied, even though they were not. That made the member permanently diverge from the other members.

The etcd team published a detailed postmortem:

• Documentation/postmortems/v3.5-data-inconsistency.md

The main lesson is: storage code must be boring. The two writes — data and consistent index — must always be atomic. A subtle change in transaction ordering can cause serious data loss.

This is also why etcd has a Robustness Testing Framework. It deliberately crashes etcd members to catch these kinds of bugs before they reach users.

Conclusion #

In this chapter, you learned:

• The WAL is an append-only log of every Raft entry.
• The backend database is a bbolt file that stores a snapshot of state as of a consistent index.
• The consistent index tells etcd which WAL entries are already reflected in the database.
• On restart, etcd reads the backend database and replays only WAL entries after the consistent index.
• Updating the data and the consistent index must be atomic.
• etcd-dump-logs and etcd-dump-db let you inspect both files.
• The v3.5 data inconsistency postmortem shows what happens when atomicity is broken.

Now that you understand how etcd stores data and recovers it, the next important question is: how does etcd detect when storage is corrupted? That is exactly the topic of the next chapter.

Continue to Data Integrity and Corruption Detection.

Generated by AI Codebase Knowledge Builder