Chapter 7: Security and Trust Boundaries #

Welcome back! In the previous chapter, Data Integrity and Corruption Detection, we saw how etcd uses hashes and consistent indexes to detect when data on disk is corrupted. Now we ask an even earlier question: should that data have been allowed to reach etcd in the first place?

That is the job of Security and Trust Boundaries.

What Problem Does This Solve? #

etcd stores critical data for systems like Kubernetes. That means we need to be very careful about who can talk to etcd and what they are allowed to do.

Imagine etcd is a castle.

• The castle is on private land: a private network.
• There are two guarded gates: port 2379 for clients and port 2380 for other etcd members.
• Each gate uses mutual TLS, meaning both sides must prove who they are before they can talk.
• Once someone passes the gate, they are considered trusted input.

This is the heart of etcd’s threat model: protect the boundary, and then trust what is inside.

If a security report claims “etcd is vulnerable,” the first question is: Did the attack cross a trust boundary without permission? If yes, it may be a real vulnerability. If no, it is probably a robustness bug, not a security vulnerability.

Our Central Use Case #

You have just joined a team that runs a small three-member etcd cluster. Your first tasks are:

1. Turn on mTLS so only authorized clients can use port 2379.
2. Turn on mTLS so only authorized cluster members can use port 2380.
3. Evaluate a security report:
   “An authenticated client can crash an etcd server by sending a malformed request. Is this a vulnerability?”

By the end of this chapter, you will be able to do all three.

Key Concepts #

1. Trust Boundary #

A trust boundary is the line between “trusted” and “untrusted.”

In etcd, the boundary is the network:

• The outside is anything that has not passed mTLS.
• The inside is traffic that has passed mTLS and is therefore trusted.

You can think of mTLS as a guarded gate. The gate is what keeps the outside out. Once you are inside the castle, etcd assumes you are allowed to be there.

2. Mutual TLS (mTLS) #

Normal TLS already lets a client verify that the server is really etcd. Mutual TLS goes further: the server also verifies the client’s certificate.

Each side presents a certificate signed by a trusted certificate authority (CA). If the certificate is valid, the connection is established.

In etcd:

• Port 2379 is the client boundary. Clients must present client certificates.
• Port 2380 is the peer boundary. etcd members must present dedicated peer certificates.

These are separate gates with separate badges.

3. The Network Boundary #

etcd assumes it runs in a private, isolated network segment. It should not be exposed to the public internet.

This is like building the castle on private land. The network boundary is the outer fence. mTLS is the inner gate.

The threat model also assumes that private keys and certificates are properly protected. If an attacker steals a valid certificate, that scenario is outside the threat model, because the certificate is supposed to be a trusted badge.

4. The Client-to-Server Boundary #

Clients talk to etcd on port 2379.

For example, this command writes a value:

etcdctl put app/config v1

Without mTLS, anyone who can reach port 2379 can send this request. With mTLS, the client must first prove its identity using a certificate.

Once a client request passes mTLS, etcd treats it as trusted input. That is an important design decision.

5. The Peer-to-Peer Boundary #

etcd members talk to each other on port 2380 to run Raft consensus. This is the “boardroom” boundary.

Peer traffic includes:

• Raft log entries.
• Vote requests during leader election.
• Snapshot streams.
• Heartbeats.

This traffic is very powerful. If an attacker could send fake Raft messages, they could disrupt the whole cluster. That is why peer communication uses its own dedicated peer certificates. A certificate that can authenticate a client on port 2379 is not automatically allowed on port 2380.

Data that arrives from an authenticated peer is also treated as trusted input.

6. Authentication and Authorization #

There are two different layers:

• Authentication: proving who you are. In etcd, this is usually mTLS.
• Authorization: deciding what you are allowed to do. etcd has optional role-based access control, but it is disabled by default.

The threat model treats etcd’s built-in authorization as an optional secondary control behind mTLS. Major users like Kubernetes often rely on mTLS and network isolation instead of etcd’s RBAC.

Also, watch stream revocations are eventually consistent. A stream that is already open may deliver events for a short time after a role is revoked. That is by design.

7. Other Boundaries #

There are a few more boundaries in etcd’s threat model:

• Host boundary: etcd is a statically linked Go binary. It does not load dynamic system libraries at runtime.
• Data storage boundary: etcd does not encrypt data at rest. If you need encryption on disk, use disk encryption or encrypt values in your application.
• Build and release boundary: build scripts and test tooling run only in trusted environments. Supply-chain issues in those tools are hardening concerns, not remote vulnerabilities.

These are important, but for everyday use, the two gates on ports 2379 and 2380 are the main thing to remember.

Solving the Use Case: Enable mTLS #

Let’s set up mTLS for a small etcd member.

First, tell etcd to require client certificates on port 2379 and peer certificates on port 2380:

etcd \
  --client-cert-auth \
  --cert-file=/etc/etcd/server.crt \
  --key-file=/etc/etcd/server.key \
  --trusted-ca-file=/etc/etcd/ca.crt \
  --peer-client-cert-auth \
  --peer-cert-file=/etc/etcd/peer.crt \
  --peer-key-file=/etc/etcd/peer.key \
  --peer-trusted-ca-file=/etc/etcd/ca.crt

Let’s break this down:

• --client-cert-auth means “require client certificates on port 2379.”
• --cert-file and --key-file are the server’s own certificate and private key.
• --trusted-ca-file is the CA used to verify client certificates.
• --peer-client-cert-auth means “require peer certificates on port 2380.”
• --peer-cert-file and --peer-key-file are the member’s peer certificate and key.
• --peer-trusted-ca-file is the CA used to verify peer certificates.

Now clients must use client certificates too.

etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.crt \
  --cert=/etc/etcd/client.crt \
  --key=/etc/etcd/client.key \
  put app/config v1

Expected output:

OK

If you do not provide a valid client certificate, etcd will reject the connection before the command even runs.

Here is a simple picture of the boundaries:

How to Evaluate a Security Report #

Now let’s use the threat model to evaluate the report from our use case:

“An authenticated client can crash an etcd server by sending a malformed request.”

Ask two questions:

1. Did the attacker cross a trust boundary without authentication?
2. Did the attacker exceed the privileges they already had?

In this report, the client is already authenticated. Sending a malformed request is annoying, but it does not cross the mTLS boundary. It also does not exceed the client’s existing privileges.

So this is not a security vulnerability under etcd’s threat model.

It is still a robustness defect worth fixing, but it is not a CVE or a security advisory.

Here is the decision process:

The threat model says it plainly:

Traffic that has passed client mTLS authentication is trusted input.

That means once the gate has been passed, etcd assumes the sender is a legitimate actor.

What Happens Under the Hood? #

When a client connects to etcd with mTLS, this happens in a simplified form:

Of course, real TLS has more steps, but the key idea is that both sides verify each other before any gRPC request is processed.

In the etcd source, the client side builds a TLS config when you create a clientv3 client:

// Simplified from clientv3
tlsConfig := &tls.Config{
    Certificates: []tls.Certificate{clientCert},
    RootCAs:      caPool,
}
cli, err := clientv3.New(clientv3.Config{
    Endpoints: []string{"https://127.0.0.1:2379"},
    TLS:       tlsConfig,
})

If you do not provide a valid certificate, the client will fail to connect.

On the server side, etcd creates a TLS config that requires and verifies client certificates:

// Simplified server-side TLS config
tlsConfig := &tls.Config{
    Certificates: []tls.Certificate{serverCert},
    ClientAuth:   tls.RequireAndVerifyClientCert,
    ClientCAs:    clientCAPool,
}

The gRPC server listens on port 2379 using this TLS config. Requests only reach the key-value handlers after this TLS check succeeds.

For peer communication, etcd has a separate transport layer on port 2380. The peer transport code in server/etcdserver/api/rafthttp/transport.go sends Raft messages only to known members:

// Simplified from server/etcdserver/api/rafthttp/transport.go
func (t *Transport) Send(msg raftpb.Message) error {
    peer := t.peers[msg.To]
    if peer == nil {
        return errors.New("unknown peer")
    }
    return peer.Send(msg)
}

This is why peer certificates are so important. Once two members authenticate over mTLS, the Raft messages they exchange are treated as trusted input.

The official document that defines all of this is THREAT_MODEL.md in the etcd repository. Security researchers and vulnerability scanners are expected to read it before reporting issues.

Security Checklist for Beginners #

Here is a short checklist to remember:

• Run etcd on a private network, not the public internet.
• Enable mTLS on both port 2379 and port 2380.
• Use separate certificates for clients and peers.
• Protect private keys and certificates. If they are stolen, the boundary does not help.
• Treat authenticated traffic as trusted input.
• Treat malformed requests from authenticated users as robustness issues, not vulnerabilities.
• Remember that etcd does not encrypt data at rest. Use disk encryption if needed.

Conclusion #

In this chapter, you learned:

• etcd has two main trust boundaries: port 2379 for clients and port 2380 for peers.
• Both boundaries should use mutual TLS.
• Traffic that passes mTLS is considered trusted input.
• etcd’s built-in authorization is optional and secondary to mTLS.
• A security report is valid only if it crosses a boundary unauthenticated or exceeds an authenticated actor’s privileges.
• Malformed requests from authenticated clients are robustness defects, not vulnerabilities.

Now you can configure mTLS, protect the castle gates, and evaluate security reports with confidence.

The next chapter looks at how etcd decides which features are stable and which ones are still experimental.

Continue to Feature Gates and Release Stages.

Generated by AI Codebase Knowledge Builder