Chapter 2: Durable Persistence #

In Chapter 1: Public API and Service Routing, we walked through the front door of Temporal. We learned how requests are routed to different services. But a front door is not enough. What if the server loses power in the middle of a workflow? If everything lives in memory, it’s gone. That’s why Temporal has durable persistence.

The Problem: Workflows Must Survive Crashes #

Imagine an online ordering workflow. It must:

1. Charge the customer.
2. Tell the warehouse to ship.
3. Send a confirmation email.

Now imagine Temporal crashes after step 1 but before step 2. What should happen? The customer already paid, so the workflow must not start over and charge them again. It also must not forget that the payment happened. The safe solution is to write every important step down in durable storage before moving on.

Temporal does this for you. Workflow state, history events, and task queue data are persisted to a database or similar durable store. This is like a bank vault: every important fact is written in a ledger, and the ledger survives crashes.

A Concrete Example: OrderWorkflow #

Let’s look at a small workflow called OrderWorkflow. In the Temporal SDK, you write workflows in normal code.

func OrderWorkflow(ctx workflow.Context, orderID string) error {
    err := workflow.ExecuteActivity(ctx, ChargeCustomer, orderID).Get(ctx, nil)
    if err != nil { return err }

    return workflow.ExecuteActivity(ctx, ShipOrder, orderID).Get(ctx, nil)
}

This function says: first charge the customer, then ship the order. You do not need to mention persistence. Temporal automatically persists the workflow before and after each step.

You can start this workflow with the CLI:

temporal workflow start --task-queue orders --workflow-id order-123

Input: the command line has the workflow ID order-123 and task queue orders.

Output: the CLI prints a RunID like 8f4f.... That RunID is a unique ID for this workflow execution. The server has already durably saved the fact that this workflow started.

Later, even after a service crash, you can look at the workflow’s saved history:

temporal workflow show --workflow-id order-123

Example output (simplified):

1 WorkflowExecutionStarted
2 WorkflowTaskCompleted
3 ActivityTaskScheduled(ChargeCustomer)
4 ActivityTaskCompleted
5 ActivityTaskScheduled(ShipOrder)

These lines are the “ledger”. They show that the payment activity finished and that the next step is to schedule shipping. If the server crashed after event 4, it can resume by reading event 4. It won’t call ChargeCustomer again.

Key Concepts #

Let’s unpack the main pieces of durable persistence.

1. Workflow State #

Workflow state is the current “snapshot” of a workflow. It includes things like:

• Which activity is running.
• What has already completed.
• What timers are pending.

The state is called mutable state because it changes as the workflow progresses. It gives Temporal a fast way to know where a workflow is right now.

2. History Events #

History events are the full story of a workflow. Temporal uses an event-sourcing model: instead of only storing the latest state, it appends events like a journal.

Examples:

• WorkflowExecutionStarted
• ActivityTaskScheduled
• ActivityTaskCompleted
• SignalReceived

These events are append-only. Once written, they are part of the ledger. If the current state is ever lost, Temporal can replay these events to rebuild the state.

3. Task Queue Data #

Workflows don’t run by magic. The server needs to remember that a task is waiting for a worker. Task queue data is also persisted so that tasks are not lost if the service responsible for matching tasks crashes.

Think of a delivery driver: the order ticket is on the board. If the restaurant owner restarts the server, the ticket should still be on the board when the next driver arrives.

4. Transactions and Committed State #

A transaction is a group of writes that either all happen or none happen. Temporal uses transactions to avoid half-written facts.

For example, when an activity finishes, Temporal needs to:

• Append an ActivityTaskCompleted history event.
• Update the workflow state to show “this activity is finished.”

Those two writes are done together. If the server crashes before the transaction is committed, neither write is saved. The workflow remains at its last committed state.

The Vault Has Many Slots: Multiple Databases #

Temporal doesn’t force you to use one specific database. The server code is written against a persistence abstraction. The go.mod file in the Temporal server repository shows database driver imports for several supported backends.

require (
    github.com/go-sql-driver/mysql v1.9.3 // MySQL
    github.com/gocql/gocql v1.7.0        // Cassandra
    github.com/jackc/pgx/v5 v5.10.0       // PostgreSQL
    modernc.org/sqlite v1.51.0            // SQLite
)

These are just a few of the drivers in go.mod. They are like different kinds of vaults. You can choose the one that fits your environment:

• PostgreSQL / MySQL are common SQL choices.
• Cassandra is used when you need horizontal scaling.
• SQLite is very useful for local development.

Your application code does not care which one is used. That is the beauty of an abstraction layer.

Under the Hood: Saving an Event #

Let’s trace what happens when ChargeCustomer finishes and Temporal records the result.

Here is the same story, one step at a time:

1. The worker that ran ChargeCustomer reports back to the frontend service.
2. The frontend asks the history service to record this completion.
3. The history service opens a transaction in the database:
   • Append the ActivityTaskCompleted history event.
   • Update the workflow’s mutable state.
4. When the transaction commits, the event is durable.
5. The frontend asks the task queue store to create a new workflow task.
6. A worker picks up that task and continues the workflow.

If the server crashes at step 3 before the transaction commits, no one will know the activity completed. If it crashes after commit, the saved event tells Temporal to move on.

Diving Into the Code #

In the Temporal server repository, the persistence interface lives in a place like common/persistence/data_interfaces.go. The actual interface is much larger, but here is the general idea.

type ExecutionStore interface {
    StartWorkflowExecution(ctx context.Context, req *StartRequest) error
    UpdateWorkflowExecution(ctx context.Context, req *UpdateRequest) error
    GetWorkflowExecution(ctx context.Context, executionID string) (*ExecutionState, error)
}

This interface says: “Any database adapter must be able to start a workflow, update it, and get it.” It does not say “you must use SQL” or “you must use Cassandra.” Each database has its own adapter.

For SQL databases, an adapter might look like this (simplified from common/persistence/sql/execution_store.go):

func (s *SQLExecutionStore) UpdateWorkflowExecution(ctx context.Context, req *UpdateRequest) error {
    return s.db.WithTx(ctx, func(tx *Tx) error {
        tx.AppendHistoryEvent(req.NewEvent)
        tx.UpdateMutableState(req.NewState)
        return nil
    })
}

Notice that both changes go through the same transaction. That is how the “bank vault” works. The history event and the updated state are committed together. If one fails, both fail. The real Cassandra adapter works differently under the hood, but the promise to the rest of Temporal is the same: a durable, committed update.

What This Means for You #

As a Temporal application developer, you do not need to open a database, write SQL, or design tables. You just:

• Define your workflow with normal code.
• Use Temporal’s SDK to start activities.
• Let the server persist the details.

When an activity completes, its result is stored in history. When the workflow is retried after a crash, replaying events lets Temporal know exactly where it stopped. It can skip work that already completed and continue from the last committed state.

Conclusion #

Durable persistence is the reason Temporal workflows can live for days, months, or years. It keeps a safe, transactional ledger of workflow history and state. The server supports multiple databases, but the abstraction is the same: every important fact is written down before the workflow moves forward.

You learned:

• Durable persistence means workflow state, history events, and task queue data are not lost.
• Temporal uses transactions to make writes safe.
• The server has adapters for PostgreSQL, MySQL, Cassandra, and SQLite.
• On crash, a workflow resumes from its last committed state.

Temporal’s persistence is the vault that holds the workflow ledger. Next, we need to explore how tasks get from the vault to the worker. That is the job of task queues and matching.

Up next: Chapter 3: Task Queues and Matching.

Generated by AI Codebase Knowledge Builder