Chapter 2: etcdctl / etcdutl Command-Line Tools #

Welcome back! In the previous chapter, gRPC Key-Value API and clientv3, we wrote a small Go program to save and read a key. That is a great way to understand how applications talk to etcd. But sometimes you just want to type one command and see what is stored in the cluster. You do not want to write a Go program every time.

That is exactly where etcdctl and etcdutl come in. These are command-line tools that let you talk to etcd quickly. Think of them as your administrator’s remote control and your maintenance toolkit.

The Big Picture: Live vs Offline #

etcdctl is the standard command-line client for a running cluster. You can use it to put, get, watch, manage leases, and inspect members over the network.

etcdutl works offline on data files. You can use it to restore snapshots, defragment the database, check hashes, and inspect storage buckets. It does not talk to a running cluster. It opens the files on disk directly.

A useful analogy:

• etcdctl is like using a banking app. You log in to the live system, check your balance, and make transactions over the network.
• etcdutl is like going into the bank vault with an auditor. The vault is locked, the servers are offline, and you are inspecting physical records.

There are also two smaller companion tools: etcd-dump-logs and etcd-dump-db. They give you even deeper access to etcd’s log files and database files for forensic debugging.

Our Central Use Case #

Let’s keep things concrete. We will:

1. Start a local etcd cluster.
2. Use etcdctl to store a configuration value.
3. Use etcdctl to read it back.
4. Watch the key change.
5. Save a snapshot of the data.
6. Use etcdutl to inspect that snapshot.

This will show you when to use each tool and why.

Key Concepts #

1. etcdctl: The Live Cluster Client #

etcdctl is the command-line client that talks to a running etcd cluster over the network. It uses the same gRPC API we saw in Chapter 1. You will probably use it every day.

Common etcdctl commands:

• put — store a value under a key.
• get — read a value.
• del — delete a key.
• watch — wait for changes to a key.
• lease — manage short-lived keys.
• member — inspect or update cluster members.

Here is a simple example:

etcdctl put dashboard/color "blue"

If etcd is running on your machine, the output is:

OK

To read it back:

etcdctl get dashboard/color

Output:

dashboard/color
blue

The first line is the key, the second line is the value. This output format is simple and easy for humans to read.

If your cluster is not on localhost:2379, you can tell etcdctl where to find it:

etcdctl --endpoints http://10.0.0.5:2379 get dashboard/color

2. etcdutl: The Offline Data File Utility #

etcdutl works directly on etcd data files while etcd is not running. It is especially useful when:

• You need to restore a cluster from a snapshot.
• You need to defragment a database that has grown too large.
• You need to verify the integrity of a snapshot file.
• You want to inspect raw buckets in the database.

Common etcdutl commands:

• snapshot restore — create a new etcd data directory from a snapshot.
• snapshot status — show info about a snapshot file.
• defrag — shrink a database file.
• hashkv — compute a hash of keys and values.
• list-bucket / iterate-bucket — inspect raw database buckets.

For example, after saving a snapshot called snapshot.db, you can inspect it with:

etcdutl snapshot status snapshot.db

Output:

cf1550fb, 3, 3, 25 kB

That line shows the snapshot’s hash, revision, total keys, and size. Not bad for one command!

3. Companion Tools: etcd-dump-logs and etcd-dump-db #

The etcd repository also ships two smaller tools for deeper debugging.

• etcd-dump-logs reads the Write-Ahead Log (WAL) and prints historical log entries. It answers questions like: “What changes were proposed to the cluster and in what order?”
• etcd-dump-db inspects the backend database file directly. It can list storage buckets, iterate key-value pairs, and compute the database hash.

These tools are like security cameras inside the vault. When something goes wrong, they help you see exactly what happened.

Solving the Use Case, Step by Step #

Let’s go through the whole process together.

Step 1: Start etcd #

Open a terminal and run:

etcd

Wait until you see log messages saying that etcd is listening on port 2379.

Step 2: Put a value with etcdctl #

Open a second terminal and run:

etcdctl put dashboard/color "blue"

You should see:

OK

The key dashboard/color now has the value "blue".

Step 3: Get the value #

Run:

etcdctl get dashboard/color

You should see:

dashboard/color
blue

This is exactly the same as the Put and Get operations we wrote in Go in Chapter 1. But now we typed them instead of writing code.

Step 4: Watch for changes #

Watch is one of the coolest features of etcd. It lets you wait for changes to a key.

etcdctl watch dashboard/color

The command will block and wait. Now, in a third terminal, update the value:

etcdctl put dashboard/color "green"

In the watch terminal, you will see something like:

PUT
dashboard/color
green

Press Ctrl+C to stop watching.

Step 5: Save a snapshot #

A snapshot is a consistent, point-in-time copy of the etcd data. You can take a snapshot while the cluster is running using etcdctl:

etcdctl snapshot save snapshot.db

You will see a log message saying the snapshot was saved to snapshot.db. If you run ls, you should see the file.

Step 6: Inspect the snapshot with etcdutl #

Now we switch to the offline tool. Make sure etcd is still running, but etcdutl does not need the network. Run:

etcdutl snapshot status snapshot.db

Output:

cf1550fb, 3, 3, 25 kB

Let’s decode that output:

• cf1550fb — a short hash of the snapshot. It helps verify that different copies of the snapshot are identical.
• 3 — the revision that was snapshotted.
• 3 — the total number of keys in the snapshot.
• 25 kB — the size of the database file.

If you prefer a cleaner table, use the table output format:

etcdutl --write-out=table snapshot status snapshot.db

Output:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| cf1550fb |        3 |          3 | 25 kB      |
+----------+----------+------------+------------+

Much nicer!

Step 7: Restore from the snapshot #

etcdutl can also create a fresh etcd data directory from a snapshot. This is how you recover a cluster after a disaster.

etcdutl snapshot restore snapshot.db \
  --name member1 \
  --initial-cluster member1=http://127.0.0.1:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380

This creates a new data directory called member1.etcd with the snapshot data inside it. You could then start an etcd member using:

etcd --name member1 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --data-dir member1.etcd

This is an excellent recovery workflow: save a snapshot, restore it into a new data directory, and start a new cluster member.

What Happens Under the Hood? #

Now that you have seen the tools in action, let’s look at what happens inside.

etcdctl talks through clientv3 and gRPC #

Remember from Chapter 1 that clientv3 is the official Go client. etcdctl is really just a friendly wrapper around that same client.

When you run:

etcdctl put dashboard/color "blue"

This happens:

1. etcdctl parses your command and arguments.
2. It creates a clientv3 client using the endpoints you provided.
3. It calls the Put method, exactly like the Go code in Chapter 1.
4. clientv3 serializes a protobuf request and sends it over gRPC to the etcd server.
5. The etcd server stores the value and replies.
6. etcdctl prints the response to your terminal.

Here is a simple sequence diagram:

This is why etcdctl and the Go clientv3 examples feel so similar. They use the same underlying API.

If you open the etcd source code, you will find etcdctl’s command implementations in files like:

• etcdctl/ctlv3/command/put.go
• etcdctl/ctlv3/command/get.go
• etcdctl/ctlv3/command/watch.go

The code is more involved than this, but conceptually it looks like:

// etcdctl/ctlv3/command/put.go (very simplified)
func putCommandFunc(cmd *cobra.Command, args []string) {
    c := mustClientFromCmd(cmd)
    err := c.Put(ctx, args[0], args[1])
    if err != nil {
        exitWithError(err)
    }
    fmt.Println("OK")
}

etcdutl opens files directly #

etcdutl works differently. It does not send messages over the network. Instead, it opens the snapshot or database file on disk and reads it directly.

For example, when you run:

etcdutl snapshot status snapshot.db

This happens:

1. etcdutl opens snapshot.db in read-only mode.
2. It reads metadata buckets from the database.
3. It computes a hash of the database contents.
4. It prints the hash, revision, key count, and size.

Here is a sequence diagram:

The etcdutl code lives in etcdutl/. For example:

• etcdutl/etcdutl.go — main entry point.
• etcdutl/ctlv3/command/snapshot_command.go — snapshot related commands.
• etcdutl/ctlv3/command/defrag_command.go — defragmentation command.

A simplified version of the status command might look like:

// etcdutl/ctlv3/command/snapshot_command.go (very simplified)
func snapshotStatusCommandFunc(cmd *cobra.Command, args []string) {
    f := openSnapshotFile(args[0])
    hash, rev, keys, size := f.Info()
    fmt.Printf("%x, %d, %d, %s\n", hash, rev, keys, size)
}

The actual code does a bit more, but this gives you the idea. No gRPC, no network, no running cluster. Just direct file access.

Why Are There Two Tools? #

You might wonder: why not just use etcdctl for everything?

Because there is a big difference between talking to a live system and operating on its physical files.

• If the cluster is running and you want to read or write data, use etcdctl.
• If the cluster is down, or you want to recover data, or you need to inspect the raw database, use etcdutl.

Sometimes you need both. For example, you might use etcdctl snapshot save to create a backup, then use etcdutl snapshot restore to recover onto a new machine.

The two tools are complementary, just like a bank app and an auditor.

Companion Tools for Deep Forensics #

The etcd repository also includes:

• tools/etcd-dump-logs — reads WAL files and prints every log entry.
• tools/etcd-dump-db — reads the backend database file directly.

These are useful when you need answers to questions like:

• “What was the last thing etcd wrote before it crashed?”
• “Why does this database file have extra buckets?”
• “Are two database files identical?”

They are more low-level than etcdctl and etcdutl. You can find their documentation in:

• tools/etcd-dump-logs/README.md
• tools/etcd-dump-db/README.md

If etcdctl is the banking app and etcdutl is the auditor’s briefcase, then etcd-dump-logs and etcd-dump-db are the magnifying glasses and UV lights used to examine documents at the smallest level.

The File That Ties Some of This Together #

If you look at the classic etcd README, you will see a quick example that uses etcdctl:

etcdctl put mykey "this is awesome"
etcdctl get mykey

That tiny example is actually the first thing many people ever do with etcd. It is simple, but it already uses the same gRPC API and storage pipeline that powers Kubernetes and many other distributed systems.

When you run etcdctl put, the request flows through:

• the command parser,
• clientv3,
• the gRPC serializer,
• the etcd server’s Raft consensus engine,
• and finally into the backend database.

If you want to understand the Raft part, take a look at the next chapter: Raft Consensus Engine.

Conclusion #

In this chapter, you learned:

• etcdctl is a command-line client for a running cluster. It talks over gRPC and supports put, get, watch, leases, and member management.
• etcdutl is an offline tool for data files. It restores snapshots, defragments databases, checks hashes, and inspects buckets.
• etcd-dump-logs and etcd-dump-db provide low-level forensic access to WAL files and database buckets.
• etcdctl uses the same clientv3 library we saw in Chapter 1.
• etcdutl opens files directly and does not need a network connection.
• A snapshot can be saved with etcdctl and inspected or restored with etcdutl.

Now that you know how humans and applications talk to etcd, it’s time to look at what happens inside the cluster when you run those commands. How do multiple etcd servers agree on the same data? That is the job of the Raft consensus engine, and it is the perfect next step.

Continue to Raft Consensus Engine.

Generated by AI Codebase Knowledge Builder