Chapter 3: Task Queues and Matching #

In Chapter 2: Durable Persistence, we learned that Temporal writes every important workflow event into durable storage. That’s how workflows survive crashes. But a ledger can’t do work. A workflow must run code: charge a credit card, ship a package, send an email. Temporal needs a way to connect work that needs doing to a worker that can do it. That’s where task queues and matching come in.

Imagine a delicatessen. Customers don’t shout their orders from the door. They take a ticket from a dispenser. The deli worker calls the next number, and the customer steps up. In Temporal, workflows and activities take a number, and workers pick up the next ticket. The matching service is the person at the counter who makes sure each ticket is handled by exactly one worker.

A Concrete Use Case: OrderWorkflow #

Remember OrderWorkflow from the last chapter. It charges a customer and ships an order. If you are the developer, you need a worker to run that workflow and its activities. You also need a task queue so the worker knows where to pick up work.

First, start a worker:

c := client.Dial(client.Options{})
w := worker.New(c, "orders", worker.Options{})

w.RegisterWorkflow(OrderWorkflow)
w.RegisterActivity(ChargeCustomer)
w.RegisterActivity(ShipOrder)

w.Start() // Start polling the "orders" task queue

This small program is a worker. It connects to Temporal, registers your code, and starts polling the task queue named orders.

Then start a workflow:

c.ExecuteWorkflow(context.Background(), client.StartWorkflowOptions{
    TaskQueue: "orders",
    ID:        "order-123",
}, OrderWorkflow, "order-123")

You can also start the same workflow from the CLI:

temporal workflow start --task-queue orders --workflow-id order-123

The CLI will print something like:

Workflow start succeeded:
  RunID: 01234567-89ab-cdef-0123-456789abcdef

When the workflow starts, a workflow task is placed on the orders task queue. Your worker polls for it, executes the workflow code, and eventually schedules activity tasks—also on orders. The worker picks those up too.

Key Concepts #

Let’s break down the pieces.

Task #

A task is a unit of work that the Temporal server needs a worker to do. There are two kinds:

When you start a workflow, the server creates a workflow task. When a workflow needs an activity to run, the server creates an activity task.

By default, activities use the same task queue as their workflow. You can override that, but for this chapter we’ll keep everything on orders.

Task Queue #

A task queue is a named lane that tasks flow into and workers poll from. The name is just a string, like "orders". It is not a traditional message queue with a huge backlog; it is more like a routing label.

• Workflows specify which task queue to use with TaskQueue: "orders".
• Workers specify which task queue to poll with worker.New(c, "orders", ...).
• If the names don’t match, the work doesn’t get done.

Why not one giant queue for all workers? Because you often want to separate concerns. Payment workers might poll "payments", shipping workers poll "shipping", and so on. Task queues let you scale each kind of work independently.

Worker #

A worker is your application process that polls task queues and executes tasks. It registers workflow and activity functions with the SDK. When a task arrives, the SDK invokes the correct function.

Workers can have different capacities:

w := worker.New(c, "orders", worker.Options{
    MaxConcurrentWorkflowTaskExecutionSize: 10,
    MaxConcurrentActivityExecutionSize: 5,
})

This tells Temporal: “Run at most 10 workflow tasks at the same time, and at most 5 activity tasks.” The matching service uses that information to avoid overwhelming your worker.

Matching Service #

The matching service is the Temporal server component that manages task queues. It receives tasks from the frontend and holds them until a worker is ready. If you see “matching” in Temporal server code, think the dispatcher or the deli counter.

Polling and Capacity #

Workers don’t get tasks pushed to them in a random way. They poll: the worker asks “Got anything for me?” and waits. The matching service holds that poll open until a task becomes available, then hands it to exactly one worker.

This is important: if two workers are polling the same queue, only one gets a given task. The other waits for another task.

How the Use Case Flows #

Let’s trace OrderWorkflow from start to activity.

Let’s walk through it:

1. You start the workflow.
2. The frontend service asks the matching service to put a workflow task on orders.
3. Your worker was already polling orders. The matching service gives it the workflow task.
4. The worker runs your OrderWorkflow function. Inside the workflow, you call ExecuteActivity(ctx, ChargeCustomer, orderID).
5. Instead of calling ChargeCustomer directly, the SDK tells the server “I need an activity task.” The frontend asks matching to put an activity task on orders.
6. The worker polls again—or a different worker polling orders polls—receives the activity task, and executes ChargeCustomer.
7. The result flows back to the history service, where it is durably persisted.

If you ran the worker in the terminal, you would see logs like:

INF Worker started
INF Polling "orders"
INF Received workflow task workflowID=order-123
INF Charging customer orderId=order-123

What Happens Under the Hood? #

Now let’s peek inside the matching service.

Step-by-Step Walkthrough Without Code #

When a worker calls PollWorkflowTaskQueue or PollActivityTaskQueue, the matching service does the following:

1. Find the task queue manager for the queue name, for example "orders".
2. Look for a task in that manager’s in-memory buffer.
3. If there is no task, hold the poll request open and wait. This is called a long-poll.
4. When a new task arrives, decide which poller should get it. The matching service uses information about worker identity and capacity.
5. Hand the task to exactly one worker, and mark it as claimed so no other poller receives it.
6. If the worker never reports back, the task eventually times out and becomes eligible for delivery again.

A Glimpse at the Matching Code #

The matching service lives in service/matching in the Temporal server repository. The main component is the matching engine. Here is a simplified version of what happens when a worker polls:

// service/matching/matching_engine.go (simplified)
func (e *matchingEngine) PollWorkflowTaskQueue(ctx context.Context, req *PollWorkflowTaskQueueRequest) (*PollWorkflowTaskQueueResponse, error) {
    mgr := e.getTaskQueueManager(req.TaskQueue)
    task := mgr.WaitForTask(ctx) // Block until a task is available
    return task.ToPollResponse(), nil
}

The line mgr.WaitForTask(ctx) is where the matching service waits for work. If a task is already waiting, it returns immediately. If not, it blocks until one arrives or the poll times out.

Here is a simplified view of adding a task:

// service/matching/matching_engine.go (simplified)
func (e *matchingEngine) AddWorkflowTask(ctx context.Context, req *AddWorkflowTaskRequest) error {
    mgr := e.getTaskQueueManager(req.TaskQueue)
    mgr.AddTask(req.Task)
    return nil
}

When AddTask is called, the manager stores the task in memory and wakes up any poller that is waiting. This is like putting a new ticket in the dispenser and ringing the bell.

The real system also handles:

• Sticky queues so workflow tasks try to go back to the same worker that has the workflow state cached.
• Backlog tracking and metrics so you can see if workers are keeping up.
• Persistence of task queue metadata so the server recovers after a restart.

But the core idea is simple: wait for a task, match it to exactly one worker, and if that worker falls asleep, eventually dispatch the task to someone else.

Why “Exactly One Worker”? #

You might be wondering: if two workers poll the same queue, what prevents both from getting the same task? The matching service’s internal state.

When a task is handed to a worker, the matching service marks that task as dispatched. It is removed from the in-memory buffer and tied to that worker. Other pollers can’t see it anymore.

If the worker crashes, the task isn’t lost. The server notices the worker didn’t respond, and the task is dispatched again. That’s why Temporal gives you at-least-once execution semantics from the application perspective, but at any moment, only one worker is actively processing a task.

Think of the deli counter again: when the worker calls “Number 42!”, they write it down. They won’t call 42 again unless that customer doesn’t show up.

Conclusion #

Task queues and matching are the heart of Temporal’s worker scheduling. The matching service is the dispatcher that connects tasks to workers. Workers poll task queues. Each task is given to exactly one active worker, and if that worker fails, the task is retried.

You learned:

• Workflow tasks and activity tasks are two kinds of work.
• Workers poll named task queues.
• The matching service holds tasks and matches them to workers with capacity.
• The matching service uses in-memory buffers and long-polling to deliver work efficiently.
• The “exactly one worker” guarantee comes from tracking dispatched tasks.

Now that we can run tasks reliably, how do we find a workflow or inspect its results when there are thousands of workflows running? That’s the next chapter.

Continue to Chapter 4: Visibility and Search.

Generated by AI Codebase Knowledge Builder