Chapter 4: Cluster Membership and Peer Communication #

In the last chapter, Raft Consensus Engine, you learned how etcd members use Raft to agree on a shared log of decisions. But what exactly is an “etcd member”? How do members join, leave, and talk to each other? That is what this chapter is about.

Think of an etcd cluster as a partner board. Each member is a partner with a name and a vote. The board makes decisions by voting. New partners can join, old partners can leave, and each partner has a phone number so they can be reached. The phone number is called a peer URL.

Why This Matters #

Imagine you have a healthy three-member etcd cluster:

• infra1
• infra2
• infra3

Your company is growing. You want to add a fourth etcd member, infra4, without stopping the cluster. But you also do not want to make the board less stable.

If you add a new voting member too quickly, the board now needs more votes to make decisions. If the new member is not ready, the board could even get stuck. So etcd gives you a safer way: add the new member as a learner first. A learner can catch up with the cluster data, but it does not vote until you promote it.

By the end of this chapter, you will know:

• What a member is and what a peer URL is.
• How members communicate on port 2380.
• What quorum means and why adding members carefully matters.
• How to add, promote, and remove a member using etcdctl.

Key Concepts #

1. Every Member Has a Name, an ID, and a Peer URL #

An etcd member is a running etcd server that belongs to a cluster. Each member has:

The peer URL is usually on port 2380. This is the “private phone line” between members. Client traffic on port 2379 is separate.

2. Peer Communication #

Members talk to each other over their peer URLs. They mainly send Raft messages:

• Vote requests during elections.
• Log entries to replicate data.
• Heartbeats so followers know the leader is alive.
• Snapshot streams when a new member is far behind.

This traffic is special. It is not the same as ordinary client traffic. Only authorized cluster members should be able to send peer messages. That is why etcd uses separate peer certificates and mTLS for port 2380.

You can think of:

• Port 2379 as the front desk where clients ask for service.
• Port 2380 as the private boardroom where members talk honestly.

More details about these trust boundaries are in Security and Trust Boundaries.

3. Quorum: The Board Meeting Rule #

A Raft cluster makes decisions only when a quorum of members agrees.

Quorum is simply more than half:

Why does this matter? If too many members are down, the cluster stops accepting writes. It is better to fail a request than to risk two different members thinking they are the leader.

Now here is the tricky part: when you add a voting member to a three-member cluster, the cluster becomes four members and quorum goes from 2 to 3. That makes the cluster harder to satisfy. The board should be careful about adding new partners before they are ready. Learners solve this problem.

4. Learners: Members Without a Vote #

A learner is an etcd member that:

• Receives data from the leader.
• Keeps its copy of the store up to date.
• Does not vote.
• Does not count toward quorum.

Learners are safe to add because they cannot accidentally make quorum harder to reach. Once a learner has caught up, you can promote it to a full voting member. This is the safe way to grow a cluster.

5. Membership Changes Are Raft Decisions #

When you add, remove, or promote a member, etcd does not just edit a config file. The membership change is sent through Raft as a special proposal called a configuration change.

Because membership changes go through Raft, every member agrees on the exact same member list. The leader proposes the change, a quorum of voters confirms it, and then the decision is committed.

Solving the Use Case: Add a Member Safely #

Let’s walk through our use case step by step.

Step 1: Look at the Current Members #

First, list the members you already have:

etcdctl --write-out=table --endpoints=localhost:2379 member list

The output will look something like this:

+------------------+---------+--------+------------------------+------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+------------------+---------+--------+------------------------+------------------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:2380  | http://127.0.0.1:2379  |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
+------------------+---------+--------+------------------------+------------------------+

This table tells you who is already in the boardroom.

Step 2: Add a New Member as a Learner #

Now add infra4 as a learner. Use a new peer URL, say port 42380:

etcdctl member add infra4 \
  --peer-urls=http://127.0.0.1:42380 \
  --learner

The command will report something like:

Member aabbccddeeff0011 added to cluster 8a...
ETCD_NAME="infra4"
ETCD_INITIAL_CLUSTER_STATE="existing"

This output is important. It gives you the exact values you need to start infra4.

Step 3: Start the New Member #

Start etcd on the new machine or directory. The key flag is --initial-cluster-state=existing, which tells etcd: “You are joining an existing cluster, not starting a brand-new one.”

etcd --name infra4 \
  --initial-advertise-peer-urls http://127.0.0.1:42380 \
  --listen-peer-urls http://127.0.0.1:42380 \
  --initial-cluster-state existing \
  --initial-cluster infra1=http://127.0.0.1:2380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380

Don’t worry about every flag. The most important idea is that the new member knows it is joining an existing cluster, not creating a new one.

After starting, the new member will contact the others on port 2380, receive the current data, and catch up as a learner.

Step 4: Check the Learner’s Status #

Run member list again:

etcdctl member list

This time you should see infra4 with status learner:

+------------------+---------+--------+------------------------+------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+------------------+---------+--------+------------------------+------------------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:2380  | http://127.0.0.1:2379  |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
| aabbccddeeff0011 | learner | infra4 | http://127.0.0.1:42380 |                        |
+------------------+---------+--------+------------------------+------------------------+

The learner is now receiving data, but it is not voting yet.

Step 5: Promote the Learner #

Once infra4 has caught up, promote it to a full voting member:

etcdctl member promote aabbccddeeff0011

The output will say something like:

Member aabbccddeeff0011 promoted in cluster 8a...

Now the board has four voting partners. The promotion also goes through Raft, so every member agrees.

Step 6: Remove a Member Later #

If you ever need to remove a member, use member remove with its ID:

etcdctl member remove aabbccddeeff0011

The output will say:

Member aabbccddeeff0011 removed from cluster 8a...

Removing a member also requires quorum. Once removed, that member is no longer part of the board.

What Happens Under the Hood? #

Now let’s look inside etcd and see how membership changes are made.

A Code-Light Walkthrough #

When you run:

etcdctl member add infra4 --learner

Something like this happens:

1. etcdctl sends a gRPC MemberAdd request to an etcd server.
2. If that server is not the leader, it forwards the request to the leader.
3. The leader creates a special Raft proposal: ConfChangeAddLearnerNode.
4. The leader writes the proposal to its Raft log and sends it to followers.
5. Once a quorum of voting members acknowledges, the change is committed.
6. The new member is now in the cluster’s membership list, but it is only a learner.

Here is the same story as a diagram:

Inside the Code #

Membership logic lives in the etcd server code, mostly under server/etcdserver/api/membership/.

The actual code is complex, but the key idea is simple: a membership change is turned into a Raft configuration change.

Here is a simplified version of adding a learner:

// Simplified from server/etcdserver/api/membership/store.go
func (s *store) AddLearner(ctx context.Context, m Member) error {
    data := mustMarshalMember(m)
    cc := raftpb.ConfChange{
        Type:   raftpb.ConfChangeAddLearnerNode,
        NodeID: m.ID,
        Context: data,
    }
    return s.proposeConfChange(ctx, cc)
}

proposeConfChange is where the request enters the Raft layer. Once Raft commits the change, etcd updates its member list.

Removing a member looks similar:

// Simplified from server/etcdserver/api/membership/store.go
func (s *store) RemoveMember(ctx context.Context, id uint64) error {
    cc := raftpb.ConfChange{
        Type:   raftpb.ConfChangeRemoveNode,
        NodeID: id,
    }
    return s.proposeConfChange(ctx, cc)
}

How Peer Communication Is Implemented #

Peer messages are sent through a transport layer that knows each member’s peer URL. A simplified version of that code looks like this:

// Simplified from server/etcdserver/api/rafthttp/transport.go
func (t *Transport) Send(msg raftpb.Message) error {
    peer := t.peers[msg.To]
    if peer == nil {
        return fmt.Errorf("unknown peer")
    }
    return peer.Send(msg)
}

Each peer has an HTTP client pointed at the other member’s peer URL. That is how a Raft message gets from member1 to member2 on port 2380.

If a new member is very far behind, the leader sends it a snapshot instead of replaying every single log entry since the beginning. You will learn more about snapshots in WAL and Backend Snapshot Storage.

Securing Peer Communication #

Peer communication is powerful. If an attacker could send fake Raft messages on port 2380, they could interfere with elections and data replication. That is why etcd treats the peer boundary differently from the client boundary.

The etcd Threat Model explains this clearly:

• Port 2379 is for clients and requires client mTLS.
• Port 2380 is for peers and requires dedicated peer certificates.
• Data arriving from an authenticated peer is trusted input.

You can think of port 2380 as a keypad-locked door. Only board members with the right badge can enter.

More on this in Security and Trust Boundaries.

Examples From Real etcd Deployments #

You do not need to set up a production cluster to see membership in action. The etcd repository includes helpful examples.

Local Cluster Guide #

The local cluster guide shows how to start a multi-member cluster using a Procfile. It brings up three members named infra1, infra2, and infra3. The Procfile even includes comments about how to add a learner node.

This is a great playground for testing member add, member list, and member promote.

Kubernetes Deployment Example #

There is also an etcd on Kubernetes example. It creates etcd as a set of pods and services:

• etcd0
• etcd1
• etcd2

This shows how multiple etcd members can be deployed as separate network services while forming one logical cluster.

Changelogs Teach Us About Edge Cases #

Membership sounds simple, but edge cases are subtle. The CHANGELOG-3.6 file records fixes like:

• “MemberUpdate implicitly and unexpectedly promotes a learner”
• “Fixed an issue preventing adding a new member when one member was down, even though quorum was still satisfied”

These small fixes matter. They are also why etcd has a Robustness Testing Framework that exercises membership changes under crashes and partitions.

Conclusion #

In this chapter, you learned:

• An etcd member has a name, an ID, a peer URL on port 2380, and a client URL on port 2379.
• Members use peer communication to exchange Raft messages.
• Quorum is more than half of the voting members, and it protects the cluster from making conflicting decisions.
• A learner is a member that catches up before it gets a vote.
• You can add a member with etcdctl member add --learner, promote it with member promote, and remove it with member remove.
• Membership changes are Raft configuration changes, not just config file edits.
• Peer communication must be secured with separate mTLS certificates.

Now that you understand how members join and talk, let’s look at how they remember everything across restarts. How does an etcd member store its Raft log and snapshots on disk?

Continue to WAL and Backend Snapshot Storage.

Generated by AI Codebase Knowledge Builder