Chapter 7: Cluster Membership and Internal RPC #

In Chapter 6: Observability and Telemetry, we learned how to monitor a Temporal server and see what is happening inside it. But so far, we have talked about Temporal as if it were a single server. In real life, Temporal runs as a cluster of multiple nodes. That raises a big question: when there are many nodes, how do they agree on who owns which workflows? And how do they talk to each other?

Think about a busy library. You don’t have just one librarian. You have a group of librarians, each responsible for a section of the shelves. When a new librarian joins, everyone needs to update their mental map so they know who is responsible for which section. If a patron asks for a book, the front desk needs to know which librarian to call.

In Temporal, that shared map is called cluster membership. The phone call to the librarian is called internal RPC. Let’s open this up and see how it works.

The Problem: Many Nodes Must Work Together #

A Temporal cluster is a set of nodes that coordinate ownership of workflows. A client might send a request to Node A. But the workflow’s history might be stored by Node B. How does Node A know that? And how does Node A ask Node B to do something?

Without this system, a multi-node cluster would be chaos. Every node would need to store every workflow, or reply “I don’t know” all the time. Temporal solves this with two libraries:

• ringpop-go provides a consistent hash ring for membership and sharding.
• tchannel-go supplies internal RPC between nodes.

This is like having a shared library map and an intercom system.

A Concrete Use Case: Starting a Workflow in a 3-Node Cluster #

Imagine you run a Temporal cluster with three nodes:

• node-a
• node-b
• node-c

A client sends a request to start a workflow. The workflow ID is order-123.

temporal workflow start \
  --task-queue orders \
  --workflow-id order-123

The CLI prints something like:

Workflow start succeeded:
  RunID: 9a1f... 

But what happened behind the scenes? The request reached node-a. How did node-a know which node should own this workflow? It used the consistent hash ring. Then it used internal RPC to talk to the owner.

Let’s build up the key ideas first.

Key Concepts #

1. Cluster Membership #

Cluster membership means keeping track of which nodes are alive, what their addresses are, and what role they play.

ringpop-go maintains this membership list. When a new node joins, the news spreads through the cluster. When a node stops responding, the other nodes eventually mark it as gone.

Analogy: the librarians have a whiteboard at the front desk. It says who is working today and which section each person covers. As people come and go, the whiteboard is updated.

2. Consistent Hash Ring #

A consistent hash ring is a way to decide “who owns what” using math instead of a central boss.

Imagine drawing a circle. Each node on the cluster claims a section of the circle. When you have a workflow ID, you hash it to a point on the circle. The node that owns that section is the owner of the workflow.

Analogy: the library map shows which librarian handles which shelf. A book title is looked up on the map, and the map says “Alice is responsible for this section.”

Why “consistent”? Because when a node is added or removed, only a small portion of workflows change owners. Most workflows stay with the same node. This makes the cluster stable.

3. Sharding #

The ownership is not one workflow at a time. Temporal groups workflows into shards. Each shard is a bucket of workflow state. The hash ring maps shards to nodes.

If you have 100 shards and 3 nodes, then roughly 33 shards are owned by each node. When a workflow ID is hashed, it maps to a shard, and the shard maps to a node.

Analogy: the library doesn’t hand each book to a librarian. It assigns whole sections. If a new librarian arrives, some sections move, but not every single book.

4. Internal RPC #

Once Node A knows that Node B owns the workflow, it needs to ask Node B to do something. For example, “please persist this workflow state.” That request is called an internal RPC (Remote Procedure Call).

Temporal uses tchannel-go for these internal calls. It is like using the library’s intercom system to ask another librarian for help.

Solving the Use Case #

Let’s trace what happens when you start order-123.

For this example, assume the hash ring says that order-123 belongs to a shard owned by node-b. The client’s request reaches node-a.

Here is what happens inside the cluster:

Wait, where is the client? The client is outside this diagram. It first sends StartWorkflowExecution to Node A. From Node A’s perspective:

1. Node A receives the start request.
2. Node A asks the ring: “Who owns order-123?”
3. The ring answers: node-b.
4. Node A sends an internal RPC to Node B: “Please persist this workflow.”
5. Node B persists the workflow state, as we saw in Chapter 2: Durable Persistence.
6. Node B returns the RunID to Node A.
7. Node A returns the RunID to the client.

The client never sees this. It only knows the workflow started successfully. The internal routing is invisible.

Under the Hood #

Now let’s peek at how Temporal uses these two libraries.

Ringpop for Membership and Sharding #

In the Temporal server repository, the membership code lives under common/membership. The concrete Ringpop implementation uses ringpop-go.

Here is a simplified version of what happens when a node joins the ring and looks up a workflow key:

ring := ringpop.New("temporal-cluster", nil)
ring.AddMember("node-a:6933")
ring.AddMember("node-b:6933")

owner, _ := ring.Lookup("order-123")
// owner is an address like "node-b:6933"

This code creates a ring, adds two members, and asks “who owns this key?” The Lookup method uses consistent hashing to return the address of the node that owns the shard containing order-123.

TChannel for Internal RPC #

Once Temporal knows the owner, it needs to send an RPC to that node. That’s what tchannel-go does.

Here is a very simplified version:

ch, _ := tchannel.NewChannel("temporal-internal", nil)
client := ch.GetSubChannel("history")

call, _ := client.BeginCall(ctx, "GetWorkflowState", arg)

This code creates an internal TChannel client, selects the history sub-channel, and begins a call to a method named GetWorkflowState. The arg would contain the workflow ID. TChannel handles the transport and serialization.

These two libraries appear in the Temporal server’s go.mod file:

• github.com/temporalio/ringpop-go v0.1.0
• github.com/temporalio/tchannel-go v1.22.1

They work together like the map and the intercom.

What Happens When a Node Joins or Leaves? #

When a new node joins, ringpop-go updates the ring. Some shards move to the new node. The cluster becomes healthier because the workload is spread across more machines.

When a node leaves, the ring is updated again. The shards owned by that node are reassigned to the remaining nodes. Other nodes detect that the node is gone through the membership protocol.

Analogy: when a librarian goes home sick, the front desk updates the map. The other librarians divide the sick librarian’s section among themselves. Normally, everyone agreed on who owns what before, so there is no confusion.

Conclusion #

Cluster membership and internal RPC are the glue that holds a Temporal cluster together. ringpop-go gives every node the same map of responsibilities. tchannel-go lets one node ask another node to perform work. Together, they make a multi-node Temporal cluster feel like one big virtual server.

You learned:

• A Temporal cluster uses membership to know which nodes are alive.
• A consistent hash ring decides which node owns which shard.
• Shards group workflows together.
• Internal RPC lets nodes talk to each other over TChannel.
• When nodes join or leave, the ring updates and workflows are rebalanced.

Now that we know how the cluster organizes itself, let’s zoom out even more. How do applications actually compose workflows, and what is the life cycle of a workflow from start to finish?

Continue to Chapter 8: Application Composition and Lifecycle.

Generated by AI Codebase Knowledge Builder