The Six Substrates — A Map of the Broker

The Six Substrates — A Map of the Broker #

RabbitMQ is an Erlang/OTP application. Its internal structure is a forest of supervised processes, each with a defined role. The substrate decomposition — Transport, Channel Multiplexer, Routing, Queue Storage, Delivery, Cluster/Metadata — maps directly onto this process forest. Understanding the process topology is prerequisite to understanding any individual substrate.

The OTP Supervision Tree #

At startup, RabbitMQ boots a hierarchy of supervisors. The relevant part for broker operation:

rabbit_sup (top-level)
├── rabbit_connection_sup_sup
│   └── rabbit_connection_sup (one per connection)
│       ├── rabbit_reader  ← Transport substrate
│       ├── rabbit_writer  ← Transport substrate
│       ├── rabbit_heartbeat ← Transport substrate
│       └── rabbit_channel_sup_sup
│           └── rabbit_channel_sup (one per channel)
│               ├── rabbit_channel  ← Channel Multiplexer substrate
│               └── rabbit_limiter  ← Delivery substrate
├── rabbit_vhost_sup_sup
│   └── rabbit_vhost_sup (one per vhost)
│       ├── rabbit_amqqueue_sup (one per queue)
│       │   └── rabbit_amqqueue_process  ← Classic Queue Storage + Delivery
│       └── rabbit_vhost_msg_store  ← Classic Queue Storage
├── rabbit_khepri (or Mnesia)  ← Cluster/Metadata substrate
└── rabbit_peer_discovery  ← Cluster/Metadata substrate

Quorum queues have a different process structure: they are managed by Ra (the Raft library), not supervised by rabbit_amqqueue_sup. Each quorum queue is a Ra server on each member node, supervised by Ra’s own supervision tree.

The Six Substrates and Their Process Owners #

Transport substrate — rabbit_reader is the controlling process for a TCP connection. It owns the socket, runs the recvloop/mainloop, parses incoming frames, dispatches method frames to rabbit_channel processes via message-passing, and enforces connection-level flow control. rabbit_writer is a separate process that serializes outgoing frames and writes to the socket. rabbit_heartbeat is a third process that tracks send/receive timestamps and fires heartbeat frames.

Three processes per connection, all linked under rabbit_connection_sup.

Channel Multiplexer substrate — rabbit_channel is a gen_server2 process (RabbitMQ’s enhanced gen_server with priority mailbox). One channel process per AMQP channel. It implements all channel-level AMQP methods: exchange.declare, queue.declare, queue.bind, basic.publish, basic.consume, basic.ack, basic.nack, basic.reject. It owns the publisher confirm state (rabbit_confirms) and delegates prefetch/flow control to rabbit_limiter.

Routing substrate — not a process, a pure function. When rabbit_channel handles basic.publish, it calls rabbit_exchange:route/2, which calls the appropriate exchange type module (rabbit_exchange_type_topic:route/2, etc.). The routing lookup reads from ETS tables (backed by Khepri projections) — in-memory, no process hops. Routing is the only substrate that does not own a process.

Classic Queue Storage substrate — rabbit_amqqueue_process is the queue process for classic queues. It owns a rabbit_variable_queue (VQ) state, which manages the in-memory/on-disk message split. VQ delegates to rabbit_classic_queue_index_v2 (sequence metadata on disk) and either rabbit_classic_queue_store_v2 (per-queue message body store for small messages) or the vhost-shared rabbit_msg_store (for larger messages).

Quorum Queue Storage substrate — each quorum queue is a Ra server (ra_server process) running the rabbit_fifo state machine. rabbit_channel interacts with quorum queues via rabbit_fifo_client, which sends commands to Ra (ra:pipeline_command/2) and handles Ra events (ack/nack, deliveries).

Delivery substrate — rabbit_queue_consumers is not a process; it is a data structure (an ETS-backed state) owned by the queue process (rabbit_amqqueue_process for classic, rabbit_fifo for quorum). It tracks registered consumers, their prefetch windows, and the unacked message set. rabbit_limiter is a gen_server that enforces per-consumer and per-channel prefetch limits.

Cluster/Metadata substrate — rabbit_khepri wraps the Khepri KV store (itself a Ra Raft application). All exchange, queue, binding, vhost, user, and permission definitions live here. Classic queue metadata lives in Khepri (or Mnesia on older nodes). Quorum queue metadata lives in their Ra log plus Khepri for the queue record. rabbit_db_* modules (rabbit_db_exchange, rabbit_db_binding, rabbit_db_queue, etc.) provide a backend-agnostic API that dispatches to either Mnesia or Khepri based on the khepri_db feature flag.

A Message’s Journey Through All Six Substrates #

Tracing a basic.publish from TCP arrival to consumer delivery shows how the substrates hand off to each other:

1. TCP bytes arrive at socket
   → rabbit_reader (Transport substrate)
   → recvloop reads bytes into buffer
   → handle_input assembles complete frame
   → dispatches basic.publish method + content to rabbit_channel (message send)

2. rabbit_channel receives basic.publish
   → Channel Multiplexer substrate
   → checks credits with rabbit_limiter (credit_flow)
   → calls rabbit_exchange:route/2

3. rabbit_exchange:route/2
   → Routing substrate (pure function, no process hop)
   → rabbit_exchange_type_topic:route/2
   → rabbit_db_topic_exchange:match/3 (ETS trie lookup)
   → returns [QueueName1, QueueName2, ...]

4. rabbit_channel delivers to each queue process
   → for classic queue: sends message to rabbit_amqqueue_process via gen_server2 cast
   → for quorum queue: rabbit_fifo_client:enqueue/3 → ra:pipeline_command/2

5. rabbit_amqqueue_process (Classic Queue Storage substrate)
   → rabbit_variable_queue:publish/5
   → writes to queue index (seq_id metadata)
   → writes body to classic_queue_store_v2 or msg_store (if persistent)

6. rabbit_amqqueue_process triggers delivery attempt
   → Delivery substrate: rabbit_queue_consumers:deliver/5
   → selects eligible consumer (respects prefetch, single-active-consumer)
   → rabbit_channel:deliver/4 (message send to channel process)

7. rabbit_channel sends basic.deliver to rabbit_writer (message send)
   → Transport substrate: rabbit_writer serializes frames
   → frames written to TCP socket

Seven hops, four process boundaries (rabbit_reader → rabbit_channel → rabbit_amqqueue_process → rabbit_channel → rabbit_writer), two pure function calls (routing, VQ publish).

Process Isolation and Failure Semantics #

The OTP process structure defines the failure blast radius:

• rabbit_reader crash → TCP connection closed → all channels closed → all unacked messages requeued. Does not affect other connections.
• rabbit_channel crash → channel error → connection survives → other channels unaffected. Unacked messages for this channel’s consumers requeued.
• rabbit_amqqueue_process crash (classic queue) → queue process restarts (supervisor) → if durable: recovers from index + msg_store. If transient: queue is gone, consumers cancelled.
• Ra server crash (quorum queue) → if majority survives: Raft elects new leader, queue continues. If minority: queue unavailable until quorum restored.
• rabbit_khepri / Ra crash → cluster metadata unavailable → new declarations fail, but existing queues continue delivering (metadata cached in ETS projections).

The substrate boundaries are also the fault isolation boundaries. This is not accidental — OTP’s supervision model enforces it.

Summary #

RabbitMQ’s six substrates map to distinct processes (or pure functions) in the OTP supervision tree. Transport owns the socket. Channel Multiplexer owns per-channel state. Routing is a pure function over ETS. Classic Queue and Quorum Queue Storage are separate process families. Delivery is a data structure owned by queue processes. Cluster/Metadata is a shared Ra-backed store with ETS projections for read performance. A message crosses four process boundaries between TCP arrival and consumer delivery.