Queues — Message Storage and Lifecycle

Queues — Message Storage and Lifecycle #

A queue is where messages live. Unlike an exchange (which routes and discards), a queue stores messages until they are consumed or expire. Every message that reaches a consumer passed through a queue first. Understanding queue properties, arguments, and types is essential for building reliable AMQP applications.

Declaring a Queue #

channel.queue_declare(
    queue='order-processing',
    durable=True,          # survives broker restart
    exclusive=False,       # not restricted to this connection
    auto_delete=False,     # not deleted when last consumer leaves
    arguments={
        'x-message-ttl': 86400000,         # message TTL: 24 hours (ms)
        'x-dead-letter-exchange': 'dlx',   # dead letter routing
        'x-max-length': 10000,             # max queue depth
        'x-queue-type': 'quorum',          # quorum queue (RabbitMQ 3.8+)
    }
)

queue_declare returns a method.Queue.DeclareOk response with:

• queue: the queue name (useful when auto-generating names)
• message_count: messages currently in the queue
• consumer_count: current consumer count

Core Properties #

Durable #

A durable queue survives a broker restart. The queue definition is persisted to disk. On restart, the broker re-declares the queue.

Important: queue durability does not mean message durability. A durable queue that receives non-persistent messages (delivery-mode=1) will lose those messages on restart — the queue exists, but the messages are gone. For crash-safe messaging, both the queue must be durable AND messages must be persistent (delivery-mode=2, covered in Chapter 8).

A non-durable (transient) queue exists only in memory and disappears on broker restart. Used for ephemeral work queues, RPC reply queues, and short-lived subscriptions.

Exclusive #

An exclusive queue has two properties:

1. Only the connection that declared it can access it (consume, bind, declare).
2. It is automatically deleted when the declaring connection closes.

Exclusive queues are the standard pattern for temporary, connection-scoped queues:

# Auto-named exclusive queue — broker assigns a unique name
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue  # e.g., 'amq.gen-XYZ123...'

Use cases:

• RPC reply queues (each client gets its own reply queue, deleted on disconnect)
• Per-session subscriptions (consumer’s private queue)
• Temporary work queues in tests

Attempting to access an exclusive queue from a different connection raises RESOURCE_LOCKED (reply-code 405).

Auto-Delete #

An auto-delete queue is automatically deleted when all consumers cancel their subscriptions (or the last consumer’s channel/connection closes). The queue is not deleted when messages are drained — only when all consumers have left.

channel.queue_declare(
    queue='temp-worker',
    auto_delete=True
)

Auto-delete queues that have never had a consumer are not auto-deleted — they linger until a consumer connects and then disconnects. This is a common surprise: an auto-delete queue declared before any consumer connects will remain until its first consumer leaves.

Difference from exclusive: auto-delete queues are accessible to multiple connections. Exclusive queues are restricted to one connection and auto-deleted when it closes.

Queue Arguments (x- Arguments) #

Queue behavior is extended via the arguments table. The x- prefix is convention for RabbitMQ extensions.

x-message-ttl: Message Time-To-Live #

Messages older than the TTL are expired and dead-lettered (or discarded if no DLX is configured).

channel.queue_declare(
    queue='ephemeral-tasks',
    arguments={'x-message-ttl': 300000}  # 5 minutes in milliseconds
)

The TTL is applied at the queue level — when a message’s age exceeds the queue’s TTL, it is expired. This differs from the per-message expiration property (Chapter 8), which sets a TTL on an individual message. When both are set, the shorter of the two is applied.

TTL countdown starts from when the message arrives in the queue, not from when it was published. A message that sits at the head of the queue for longer than the TTL is expired immediately on the next evaluation (lazy queues and quorum queues evaluate on consume; classic queues evaluate when the message reaches the head).

x-dead-letter-exchange: Dead Letter Routing #

When a message is:

• Rejected with basic.nack or basic.reject and requeue=False
• Expired (per-message TTL or queue TTL)
• Dropped because the queue has exceeded x-max-length (if the overflow behavior is dead-letter-on-overflow)

…it is dead-lettered: forwarded to the dead-letter exchange (DLX) instead of discarded.

# Declare DLX and its queue
channel.exchange_declare('dlx', 'topic')
channel.queue_declare('dead-letters')
channel.queue_bind('dead-letters', 'dlx', routing_key='#')

# Declare main queue with DLX
channel.queue_declare(
    queue='tasks',
    arguments={
        'x-dead-letter-exchange': 'dlx',
        'x-dead-letter-routing-key': 'tasks.dead',  # optional override
    }
)

Dead-letter routing key: by default, the original routing key is preserved when dead-lettering. If x-dead-letter-routing-key is set, that routing key is used instead. This allows the DLX to route dead letters differently from live messages.

Dead-letter headers: dead-lettered messages carry additional headers in x-death:

{
  "x-death": [{
    "queue": "tasks",
    "reason": "rejected",
    "time": "2026-03-26T10:00:00Z",
    "exchange": "work-exchange",
    "routing-keys": ["tasks"],
    "count": 1
  }]
}

Each time a message is dead-lettered, a new entry is prepended to x-death. Messages that bounce repeatedly (rejected → dead-lettered → re-queued → rejected again) accumulate x-death entries, providing a full history of rejections.

x-max-length and x-max-length-bytes #

These arguments cap the queue size:

• x-max-length: maximum number of messages in the queue.
• x-max-length-bytes: maximum total body size of messages in the queue.

When the limit is exceeded, the overflow behavior is controlled by x-overflow:

• drop-head (default): the oldest message at the head of the queue is dropped or dead-lettered.
• reject-publish: the broker rejects new publish attempts with a basic.return (if mandatory=True) or silently drops.
• reject-publish-dlx: reject new publishes and dead-letter them.

channel.queue_declare(
    queue='bounded-queue',
    arguments={
        'x-max-length': 1000,
        'x-overflow': 'reject-publish'
    }
)

reject-publish with publisher confirms (Chapter 10) enables backpressure: the broker nacks the publish when the queue is full, and the producer can slow down accordingly.

x-max-priority: Priority Queues #

A priority queue orders messages by priority rather than FIFO:

channel.queue_declare(
    queue='priority-tasks',
    arguments={'x-max-priority': 10}  # 10 priority levels (0-9)
)

When publishing, set the priority property on the message (0 = lowest, 9 = highest for a 10-level queue). The broker delivers higher-priority messages before lower-priority ones, regardless of arrival order.

Limitation: priority queues consume more memory (one sub-queue per priority level). For most use cases, 2-3 priority levels (0, 5, 10) are sufficient. Setting x-max-priority to 255 (the maximum) creates 255 sub-queues and is rarely justified.

x-queue-type: Classic vs Quorum vs Stream #

RabbitMQ 3.8+ supports multiple queue types via x-queue-type:

• classic (default): original queue type, stored in-memory with optional disk persistence. Single-master, mirrored to replicas via classic mirroring (deprecated in 3.12).
• quorum: replicated queue using Raft consensus. Recommended for high-availability and data safety requirements.
• stream: append-only log with offset-based consumption (Kafka-like semantics). Not covered in this chapter.

channel.queue_declare(
    queue='resilient-tasks',
    durable=True,
    arguments={'x-queue-type': 'quorum'}
)

Classic vs Quorum Queues #

Quorum queue key feature — delivery limit: if a message is nacked and requeued repeatedly, it can poison a worker. Quorum queues support x-delivery-limit:

channel.queue_declare(
    queue='safe-tasks',
    durable=True,
    arguments={
        'x-queue-type': 'quorum',
        'x-delivery-limit': 5,               # dead-letter after 5 nacks
        'x-dead-letter-exchange': 'dlx',
    }
)

After 5 failed deliveries, the message is dead-lettered. Classic queues require the application to track delivery counts manually.

When to use quorum queues: any durable, long-lived queue in a clustered environment. Quorum queues tolerate broker failures without message loss. The tradeoff: slightly higher latency (Raft consensus per message) and no support for exclusive/priority queues.

Queue Lifecycle #

[declared] → [receiving messages] → [delivering to consumers] → [empty]
    ↓                                        ↑
[auto-deleted when last consumer leaves] ← [all consumers cancel]
    OR
[deleted explicitly via queue.delete]
    OR
[exclusive queue's connection closes → deleted]

Deleting a queue:

channel.queue_delete(
    queue='old-queue',
    if_unused=True,   # only delete if no consumers
    if_empty=True     # only delete if no messages
)

Purging a queue (delete all messages, keep queue):

channel.queue_purge(queue='old-queue')

Checking queue status (passive declare):

result = channel.queue_declare(queue='my-queue', passive=True)
print(f"Messages: {result.method.message_count}, Consumers: {result.method.consumer_count}")

Queue Naming Conventions #

Named queues: use descriptive, stable names. Prefer service.purpose format:

orders.processing
payments.pending
notifications.email

Server-named queues: pass queue='' to let the broker generate a unique name:

result = channel.queue_declare(queue='', exclusive=True)
name = result.method.queue  # e.g., 'amq.gen-ABCD1234...'

Server-named queues always have unique names and are suitable for temporary, per-connection queues. They are always exclusive (non-sharable).

Summary #

Queues store messages until consumed. The three core properties — durable, exclusive, auto-delete — define the queue’s lifetime. Queue arguments extend behavior: TTL expires old messages, DLX routes failures, max-length caps depth, priority reorders delivery. Classic queues are flexible; quorum queues are safe for production HA deployments. Queue durability and message persistence are independent — both must be set for crash-safe messaging.