Chapter 6: Observability and Telemetry #

In Chapter 5: Archival and Object Storage, we saw how old workflow histories can be moved to cheap object storage. That keeps the primary database fast. But now imagine you are the person who runs the Temporal server. The server is processing thousands of workflows. Is it healthy? Is it slow? Is a worker broken? You need a way to see what is happening inside the server.

This is where observability and telemetry come in. Telemetry means “data about the system’s behavior”. Observability means “the ability to ask questions about that behavior”. Temporal integrates several tools to give you this ability: Prometheus, OpenTelemetry, statsd, and structured logging.

Analogy time: Imagine driving a car. Your dashboard tells you speed, fuel, and engine temperature. That’s like metrics. If something goes wrong, you might review the trip recorder to see exactly where you drove. That’s like traces. And you might keep a written log of weird engine sounds. That’s like logs. All three together tell you whether your car is healthy, and if not, why.

A Concrete Use Case: Is My Temporal Server Healthy? #

Suppose you get a page at 3 a.m.:

“Some workflows are running very slowly!”

You need to answer three questions:

1. What is happening right now? – Are requests failing? Is the CPU high? How many tasks are stuck?
2. Where in the system is time going? – Is the frontend slow? Is the database slow? Is a worker not responding?
3. What recent events led to this? – Are there errors or warnings in the logs?

With Temporal’s observability tools, you can check all three.

Let’s start with the simplest way: “open the hood” and look at metrics.

Key Concepts #

Before we dig deeper, let’s break down the main pieces.

The Three Pillars: Metrics, Traces, Logs #

Observability systems usually talk about three kinds of telemetry.

Temporal emits all three.

Metrics #

Metrics are numbers measured over time. For example:

• How many workflow tasks were completed per second?
• How many requests returned an error?
• How long does it take to start a workflow?

Metrics are often exposed as an HTTP endpoint that Prometheus scrapes every few seconds.

Traces #

Traces follow a single request through the system. When you call StartWorkflowExecution, that request travels from the frontend service to the history service. A trace shows every step, with durations. It can also show if the database was slow.

Traces are usually exported to a system like Jaeger or Zipkin using OpenTelemetry.

Logs #

Logs are discrete messages. When the server starts, when a request fails, when a worker polls, etc. Temporal uses structured logging, meaning logs are written in a structured format like JSON. This makes them easy to search.

Backends: Prometheus, OpenTelemetry, statsd #

Temporal doesn’t force you to use one tool. It supports several telemetry backends.

• Prometheus – collects metrics. You can scrape an endpoint like http://temporal-server:9090/metrics.
• OpenTelemetry – collects traces and metrics in a vendor-neutral way. It can send data to many different products.
• statsd – a simple, older protocol for sending metrics. Some operators still use it.

You can see all of these in the go.mod file of the Temporal server:

require (
    github.com/prometheus/client_golang v1.21.0
    go.opentelemetry.io/otel v1.44.0
    github.com/cactus/go-statsd-client/v5 v5.1.0
)

These are the libraries Temporal uses to talk to those backends.

Solving the Use Case: Checking Health #

Let’s step through how you would actually answer the 3 a.m. question.

Step 1: Look at Metrics with Prometheus #

If your Temporal server is configured to expose Prometheus metrics, you can open the metrics endpoint in a browser or grab it with curl:

curl http://localhost:9090/metrics

The output is a big list of text lines, each describing one metric. Here’s a very simplified example:

# HELP temporal_workflow_schedule_to_start_latency_ms How long tasks wait
# TYPE temporal_workflow_schedule_to_start_latency_ms histogram
temporal_workflow_schedule_to_start_latency_ms_bucket{taskqueue="orders",le="100"} 120
temporal_workflow_schedule_to_start_latency_ms_bucket{taskqueue="orders",le="500"} 150
temporal_workflow_schedule_to_start_latency_ms_bucket{taskqueue="orders",le="+Inf"} 160

This says: on the orders task queue, 120 workflow tasks started within 100 milliseconds, 150 started within 500 milliseconds, and 160 started in total. If your workers are slow, the +Inf bucket will keep growing while the lower buckets don’t.

Most people don’t read these lines directly. Instead, they use Grafana to turn them into charts. But the important thing is: the data is there.

Step 2: Look at Traces with OpenTelemetry #

If the metrics say “requests are slow”, you want to know why. That’s where traces help.

Start a workflow with the CLI:

temporal workflow start --task-queue orders --workflow-id order-123

While that request is happening, Temporal can generate a trace and export it to a tracing backend. In a tool like Jaeger, you might see something like:

StartWorkflowExecution 250ms
  ├── Frontend validation  10ms
  ├── History write        200ms
  └── Add task to queue    40ms

This tells you that most of the time was spent writing to history, not in network overhead. That’s a huge clue for finding the root cause.

Step 3: Look at Structured Logs #

Sometimes you need to see events. Temporal logs structured JSON lines. You can inspect them:

tail -f /var/log/temporal/simple-service.log | jq '.'

Example log line:

{
  "level": "error",
  "time": "2026-04-10T03:12:33.123Z",
  "msg": "Workflow task failed",
  "workflow_id": "order-123",
  "task_queue": "orders",
  "error": "timeout"
}

This log line tells you exactly which workflow failed, on which task queue, and why. Searchable and easy to understand.

What Happens Under the Hood? #

Now let’s see how Temporal creates all this telemetry.

Step-by-Step Walkthrough #

When a request arrives at the Temporal server, several things happen outside your view:

1. The frontend service receives the request.
2. Middleware starts a timer and creates a “span” for the trace.
3. The frontend calls the history service.
4. Each service logs important events.
5. After the response, the middleware sends metric data and trace data to the configured backends.

Here’s a visual for one request:

The most important part is the middleware. It wraps every request and automatically collects telemetry, so the workflow logic doesn’t have to.

A Peek at the Code #

In the Temporal server repository, telemetry code lives in areas like common/telemetry, common/metrics, and common/log. Let’s look at a very simplified version of what a metrics middleware might do.

First, we can emit a duration metric with Prometheus:

import "github.com/prometheus/client_golang/prometheus"

var duration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "temporal_request_duration_seconds",
    },
    []string{"service"},
)

func Instrument(service string, fn func()) {
    start := time.Now()
    fn()
    duration.WithLabelValues(service).Observe(time.Since(start).Seconds())
}

Here’s what it does:

• duration is a Prometheus histogram that tracks request latencies.
• Instrument runs the actual logic and then records how long it took.

No business logic is polluted with telemetry.

For tracing, Temporal uses the OpenTelemetry gRPC middleware. You can configure it like this:

import (
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "google.golang.org/grpc"
)

server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewClientHandler()),
)

This automatically creates spans for every incoming gRPC call. It’s like adding a small black box to every trip.

For logs, Temporal uses go.uber.org/zap to create structured logs:

logger.Info("request completed",
    zap.String("service", "frontend"),
    zap.Duration("duration", 42*time.Millisecond),
    zap.Int("status", 200),
)

The zap library turns those fields into a structured JSON line. Easy to parse and search.

Why This Matters to You #

If you’re a beginner, you may not need to configure all of this on day one. But knowing that Temporal has observability built in is powerful. When something goes wrong, you don’t need to guess. You have:

• Metrics: to notice the problem.
• Traces: to locate the problem.
• Logs: to understand the details.

The car analogy again: metrics are the red warning light. Traces are the mechanic’s diagnostic tool. Logs are the mechanic’s notepad. Together, they let you fix the car without taking it apart blindfolded.

Conclusion #

Observability and telemetry turn a hidden, complex server into a transparent system you can inspect, measure, and debug.

You learned:

• Metrics, traces, and logs are the three pillars of observability.
• Temporal integrates Prometheus, OpenTelemetry, statsd, and structured logging.
• Metrics tell you what is happening, traces tell you where, and logs tell you why.
• The server emits all this automatically through middleware and logging libraries.

In the next chapter, we’ll climb higher up the system and look at how multiple Temporal servers work together as a group. That’s where cluster membership and internal RPC come into play.

Continue to Chapter 7: Cluster Membership and Internal RPC.

Generated by AI Codebase Knowledge Builder