Chapter 3: Raft Consensus Engine #

Welcome back! In the previous chapter, etcdctl / etcdutl Command-Line Tools, you learned how to talk to a running etcd cluster using command-line tools. You ran etcdctl put, saw OK, and read the value back.

But have you ever wondered: how do multiple etcd servers agree on that value?

If you have only one etcd server, it can just write the value to disk. But etcd is built to run as a cluster of multiple servers. If those servers all wrote independently, they would quickly disagree. One server might have "blue", another might have "green", and a third might not have the key at all.

That is where Raft comes in.

Raft is the distributed consensus algorithm that makes etcd reliable. It is the engine that makes sure a cluster of etcd servers behaves like one etcd server.

The Problem: Getting Multiple Servers to Agree #

Imagine you and two friends run a small club. You keep a shared meeting notebook.

• If everyone writes in the notebook at the same time, pages get messy.
• If one person writes something and nobody else sees it, the club does not really agree on the decision.
• If the notebook owner goes home, the rest of the club should still know what was decided.

Raft solves this by choosing one person to be the leader. The leader writes each decision into the notebook. The others copy it. A decision is official only after most of the club has confirmed they have it.

In etcd, “the club” is a cluster of etcd servers, and “the notebook” is the Raft log.

Our Use Case for This Chapter #

Let’s keep things simple and concrete.

You have a three-member etcd cluster:

• member1
• member2
• member3

You run:

etcdctl put app/config "v1"

How do all three members decide that app/config is now "v1"?

By the end of this chapter, you will understand exactly what happens, both at a high level and inside the code.

Key Concepts #

Before we walk through the full example, let’s look at the building blocks.

1. Consensus #

Consensus means that a group of servers agrees on the same data, even if some servers fail or messages are delayed. Raft is a consensus algorithm, which is a clear set of rules for reaching that agreement.

You can think of Raft as a “captain election” plus a “shared log” system.

2. The Raft Log #

Raft does not just store the final value. It stores a log of commands.

A log entry might look like:

Entry 1: set app/config = "v1"
Entry 2: set app/config = "v2"
Entry 3: delete app/config

Every server starts with the same initial state and applies the same entries in the same order. This is called a replicated state machine.

It is like every club member having the same page of meeting minutes. If everyone reads the minutes in order, everyone reaches the same understanding.

3. Leader Election #

Raft organizes time into terms. In each term, servers try to elect a leader.

• Most of the time, a server is a follower.
• If a follower does not hear from a leader for a while, it becomes a candidate.
• The candidate asks the other servers to vote for it.
• If a candidate gets votes from a majority, it becomes the leader for that term.

Think of a class electing a student to take notes. If the note-taker disappears, the class holds a new election.

4. Quorum and Majority #

A majority, also called a quorum, is more than half of the cluster.

For a cluster size N:

• If N = 1, quorum is 1.
• If N = 3, quorum is 2.
• If N = 5, quorum is 3.

Raft only commits a change when a majority of members have stored it in their logs. That way, even if one member crashes, the remaining members still have the decision.

5. Linearizable Writes #

etcd promises linearizable writes.

That means: once a write returns success, a later read from any client must see that write (or a later one). The write cannot silently disappear.

Raft makes this possible by acknowledging a write only after it is committed by a majority.

If a server crashes after a successful write, the new leader will still have the log entry, because the new leader must have been part of the majority.

6. Replicated State Machine #

A state machine is just a system with a current state that changes when you give it commands.

In etcd, the state machine is the key-value store. A command like set app/config = "v1" changes the state.

Because every etcd server applies the same log entries in the same order, every server reaches the same state. Even if a server crashes and restarts, it replays the log and catches up.

Solving the Use Case: A Put in a Three-Member Cluster #

Now let’s see exactly what happens when you run:

etcdctl put app/config "v1"

The cluster has three members. Let’s say member1 is the leader.

What happens step by step #

1. etcdctl sends the put request over gRPC to an etcd server. In our example, it reaches member1.
2. member1 is the leader, so it accepts the write and appends the command to its Raft log.
3. member1 sends the log entry to member2 and member3.
4. Each follower stores the entry in its own Raft log and replies “I have it.”
5. Once member1 receives acknowledgements from enough followers to make a majority, it commits the entry.
6. member1 and the followers apply the committed entry to their key-value stores.
7. member1 sends the success response back to etcdctl.

In a three-member cluster, only one follower acknowledgement is enough, because the leader plus one follower equals two members, which is a majority. In our diagram below, both followers happen to acknowledge.

If the write succeeds, the output is:

OK

But what if the cluster does not have a majority? For example, if member2 and member3 are unreachable, member1 alone cannot commit. The command will time out, and you will get an error. etcd would rather fail the request than pretend the write is safe when it is not.

What Happens Under the Hood? #

Now let’s open the hood and look at how etcd uses Raft internally.

The two main layers #

etcd has two important layers:

1. The etcd server layer
   This layer handles gRPC requests from clients. It receives a Put, and asks Raft to replicate it.

2. The Raft layer
   This layer handles leader election, log replication, and committment.

The Raft layer is a separate Go module called go.etcd.io/raft/v3. It is a generic Raft implementation that etcd embeds. It contains no etcd-specific key-value code.

You can learn more about the module layout in Multi-Module Go Repository.

When a Put arrives at the etcd server #

The gRPC server receives a PutRequest. It creates a Raft proposal from the request data and passes it to the Raft node.

Here is a very simplified version of that code path:

// server/etcdserver/raft.go (simplified)
func (s *EtcdServer) propose(ctx context.Context, data []byte) error {
    // data is the serialized PutRequest.
    // s.r is the Raft node from go.etcd.io/raft/v3.
    return s.r.Propose(ctx, data)
}

Propose tells the Raft node: “Please put this command into the Raft log.”

If the server receiving the request is not the leader, the Raft node will forward the proposal to the leader using the peer communication protocol on port 2380.

When Raft commits the entry #

Once a majority has acknowledged the entry, the Raft layer tells etcd that the entry is committed. Then etcd applies the entry to its key-value store.

A simplified version of that apply loop looks like this:

// server/etcdserver/apply.go (simplified)
func (s *EtcdServer) applyEntries(ents []raftpb.Entry) {
    for _, ent := range ents {
        if ent.Type == raftpb.EntryNormal {
            s.applyEntry(ent)
        }
        // Remember how far the store has been updated.
        s.consistentIndex.Set(ent.Index)
    }
}

The applyEntry function decodes the PutRequest from the entry and writes the key-value pair into the backend database.

Inside the Raft library #

The Raft library itself is all about messages, votes, and log entries. It is quite complex, but you can understand its job with a small mental model.

For example, when a Raft node becomes leader, it sends an empty log entry to followers just to be sure it can reach a majority. This is called a “no-op” entry.

// go.etcd.io/raft/v3/raft.go (mental model, simplified)
func (r *raft) becomeLeader() {
    r.state = StateLeader
    // Tell followers that I am the leader.
    for _, peer := range r.peers {
        r.sendAppend(peer)
    }
}

And when a follower does not hear from a leader, it starts an election:

// go.etcd.io/raft/v3/raft.go (mental model, simplified)
func (r *raft) checkElectionTimeout() {
    if r.state != StateLeader {
        r.electionElapsed++
        if r.electionElapsed > r.electionTimeout {
            r.startElection()
        }
    }
}

The real code is more detailed, but this is the rhythm of Raft: leaders send heartbeats, followers wait, and if the leader disappears, a new election starts.

Why Peer Port 2380 Is Special #

etcd has two main network ports:

• 2379 for client requests using the gRPC API.
• 2380 for server-to-server peer communication, which carries Raft messages.

Raft messages are extremely powerful. A peer that can inject fake Raft messages could change the cluster’s mind about who the leader is or what data is committed. That is why peer communication on port 2380 is protected by its own mTLS boundary.

This means:

• A client with permission to read and write keys on port 2379 is not automatically allowed to join the Raft peer network.
• Peer certificates are separate and dedicated to cluster members.
• Raft traffic is trusted only after a peer proves its identity using those certificates.

You can think of port 2379 as the front desk where clients ask for services, and port 2380 as the private room where the members of the board make the actual decisions. Only board members with the right badge can enter that room.

More details are in Security and Trust Boundaries.

How Robustness Testing Helps Raft #

Raft is designed to survive crashes, network partitions, and slow machines. But theory is not enough. etcd needs to prove that Raft is working correctly.

That is where the robustness testing framework comes in.

Robustness tests do things like:

• Kill an etcd member without warning.
• Pause a member for a long time.
• Cut the network between members.
• Reconnect the network and watch what happens.

Then the tests compare the actual behavior of the cluster against etcd’s guarantees. They check that:

• A write that was acknowledged is still present later.
• Revisions never go backward.
• Watches do not miss events or go back in time.

These tests catch real bugs. They are described in detail in Robustness Testing Framework.

The key idea is simple: Raft should never break consensus, even when everything around it breaks.

The Role of the WAL and Snapshots #

Raft keeps its log in memory, but for crash recovery, etcd writes the log to disk too.

The Write-Ahead Log (WAL) stores Raft log entries before they are applied. If an etcd member crashes and restarts, it reads the WAL, replays the entries, and reconstructs the state.

From time to time, etcd also creates a snapshot. A snapshot is a compact copy of the state at a certain revision. This prevents the log from growing forever.

Those details are covered in WAL and Backend Snapshot Storage.

For now, just remember: Raft’s decisions live in a log, and that log is made durable by the WAL layer.

Conclusion #

In this chapter, you learned that:

• Raft is the distributed consensus algorithm that etcd uses to stay reliable.
• Servers elect a leader, and the leader proposes changes as log entries.
• A change is committed only after a majority of members acknowledge it.
• The Raft log creates a replicated state machine, where every member applies the same commands in the same order.
• etcd returns success for a write only after Raft has committed it.
• Peer communication on port 2380 is a separate, protected trust boundary.
• Robustness testing verifies that Raft keeps its guarantees under crashes and partitions.

Now you know what happens inside etcd when you run etcdctl put. But how do members join, leave, and communicate with each other as a cluster? That is exactly the next topic.

Continue to Cluster Membership and Peer Communication.

Generated by AI Codebase Knowledge Builder