Pub/Sub Notification System Analysis Note #

This note captures the full step-by-step analysis for a fanout-style pub/sub notification system: topic publication, subscriber relationship state, delivery fanout, retry/ack lifecycle, and shard ownership.

Step 1 — Normalize #

Assume the baseline prompt is:

• design a pub/sub notification system
• publishers publish events to topics
• many subscribers receive the event
• fanout is the core model
• subscribers may be online/offline
• system should scale across nodes

Normalize into state-affecting paths.

Notes on normalization:

Important choices:

• publish is append event
  • publication is an immutable fact
• subscribe is append event
  • creates subscription relationship
• unsubscribe is state transition
  • subscription lifecycle changes
• fanout is async process
  • internal expansion from topic event to per-subscriber delivery
• ack/retry are delivery-lifecycle transitions
• routing/ownership are explicit because this is distributed infra

This system is not a replayable partition log by default. It is:

• topic publication
• subscriber relationship state
• delivery/fanout lifecycle

Step 2 — Critical Path Selection #

Baseline critical paths:

Main C1 paths:

• P1 publish notification
• P2 subscribe
• P3 unsubscribe
• P4 fanout to subscribers
• P5 ack delivery
• P6 retry undelivered delivery
• P7 route to shard owner
• P8 reassign shard ownership

This system is driven by:

• immutable publication events
• authoritative subscription set
• per-subscriber delivery lifecycle
• fanout expansion correctness
• shard ownership/failover

Step 3 — Primary State Extraction #

For a fanout pub/sub system, the minimal primary state is published events, subscription edges, delivery records, and shard ownership/routing state.

Important modeling choices:

PublishedEvent #

Primary because:

• publication is immutable truth

SubscriptionEdge #

Primary because:

• it defines who should receive future events
• subscribe/unsubscribe correctness depends on it

DeliveryRecord #

Primary because:

• fanout creates per-subscriber delivery lifecycle state
• retry/ack semantics depend on it

PartitionOwnership #

Needed because:

• one shard owner should orchestrate publication/fanout for that shard

PartitionMap #

Needed because:

• routing of publication and subscription operations must be consistent

Minimal strict primary set:

• PublishedEvent
• SubscriptionEdge
• DeliveryRecord
• PartitionOwnership
• PartitionMap

Step 4 — Hard Invariants #

For a fanout pub/sub system, the hard invariants are about immutable publication, authoritative subscription state, valid delivery lifecycle, and exclusive shard ownership.

What matters most:

1. Subscription edge uniqueness #

No duplicate active subscription edge for the same subscriber/topic scope.

2. Delivery accounting #

Fanout correctness depends on a well-defined subscription cut:

• “active at publish time” is the usual baseline

3. Per-subscriber delivery lifecycle #

Ack and retry must operate on current delivery attempt state, not stale tokens.

Step 5 — Execution Context #

For the fanout baseline:

Derived implications:

• holder_may_crash = true

  • subscribers can fail while handling delivery attempts

• cross_service_write = false

  • baseline keeps publication, subscriptions, delivery state, and routing within one logical service

• bounded_staleness_allowed = false

  • strict baseline disallows stale subscription/delivery state on correctness-critical paths

• cross_service_atomicity_required = false

  • no multi-service transaction required in baseline

• exclusive_claim_required = true

  • a delivery attempt should have one active current holder/attempt token where ack/retry semantics depend on it

• guarded_by_current_state = true

  • unsubscribe, ack, retry, and failover all depend on current state

This pushes us toward:

• one authoritative owner per shard
• immutable publications
• versioned subscription state
• per-delivery lifecycle state with guarded ack/retry

Step 6 — Deterministic Mechanism Selection #

6A. Write Shape #

6B. Base Mechanism #

Why these fit:

Publish #

Publication is immutable, naturally append-only.

Subscription edge #

You can model subscribe as append-only if subscriptions are immutable edges with versioned replacement, but in the baseline practical system it is usually simpler to treat active/inactive subscription state as guarded current state.

Fanout #

Fanout is driven by immutable publication events and current subscription set. Delivery records themselves can be created append-only at first, but their lifecycle quickly becomes guarded state transitions for ack/retry.

Canonical substrate implied:

• sharded pub/sub service
• immutable publication stream per topic/shard
• authoritative subscription store
• per-subscriber delivery lifecycle state
• lease-backed shard ownership

Step 7 — Read Model / Source of Truth #

For a fanout pub/sub system, truth is mostly direct source state. Topic status views are projections.

Important point:

For correctness-critical paths:

• fanout should read authoritative subscription state
• ack/retry should read authoritative delivery state
• shard ownership/routing should be authoritative

Step 8 — Failure Handling #

What matters most:

1. Delivery dedup scope #

For a given publication and subscriber:

• create at most one current delivery record for the baseline semantic cut

2. Attempt-token fencing #

A stale subscriber ack must not complete a newer retry attempt.

3. Publication retry dedup #

If the publish API promises idempotency:

• add request-id dedup on publication

Failure summary:

This system stays correct if:

• publication is durable/idempotent as needed
• subscription state is authoritative
• delivery-record creation is deduped by (event_id, subscriber_id)
• ack/retry are guarded by current attempt token/version
• stale shard owners and stale subscriber attempts are fenced

Step 9 — Scale Adjustments #

What scales well:

This system scales by:

• partitioning topics
• partitioning subscription indexes
• letting each shard owner expand publication to delivery records for its shard scope
• batching downstream delivery

What fails first:

Usually:

• hot topic fanout explosion
• retry storms
• delivery-record storage growth
• subscription-index hot spots

Canonical design conclusion:

The mechanical outcome is:

• primary state:
  • PublishedEvent
  • SubscriptionEdge
  • DeliveryRecord
  • PartitionOwnership
  • PartitionMap
• critical invariants:
  • immutable publication truth
  • authoritative subscription set
  • one delivery record per event-subscriber baseline scope
  • guarded ack/retry with attempt fencing
  • fenced shard ownership
• mechanisms:
  • append log
  • guarded subscription and delivery transitions
  • lease-backed shard ownership
• reads:
  • direct source reads for subscriptions and delivery state
  • projections only for status/metrics

Polished interview answer:

“I’d design the pub/sub notification system as a sharded fanout service. Publication is an immutable event, subscriptions are authoritative relationship state, and fanout creates per-subscriber delivery records from the current subscription set. Acknowledgment and retry are modeled as guarded transitions on delivery records, fenced by attempt tokens so stale retries or stale acks cannot corrupt delivery state. Hot topics are handled by sharding fanout work and keeping topic status views as derived projections only.”

Concrete Substrate #

I’ll choose a service-owned sharded fanout platform as the concrete baseline, because that fits the mechanics we derived:

• immutable publication events
• authoritative subscription edges
• per-subscriber delivery records
• lease-backed shard ownership

Concrete substrate:

• pub/sub service cluster
• topics split into shards
• one leader/owner per shard
• shard leader stores:
  • publication stream for that shard
  • subscription index for subscribers in that shard scope
  • delivery records and retry schedule
• followers replicate shard mutations
• metadata layer tracks shard ownership and routing

Concrete tech family:

• service in Go or Java
• durable publication storage with RocksDB or append-only segment files
• shard replication via Raft or leader-follower replication
• metadata/control via etcd or internal quorum
• retry scheduling via shard-local time buckets or min-heaps

Operation Layer #

1. Publish notification #

API

• Publish(topic_id, payload, request_id?)

Initiator

• publisher/client

Entry point

• gateway or any pub/sub node

Authoritative decider

• current leader for target topic shard

Precondition

• valid shard routing
• optional dedup request id if publisher idempotency is required

Transition

• append PublishedEvent
• enqueue fanout work for active subscribers in that shard scope

Response

• {event_id}

Failure cases

• stale routing -> retry with updated shard map
• response loss -> duplicate publish unless request_id dedup is used

2. Subscribe #

API

• Subscribe(subscriber_id, topic_id, expected_version?)

Initiator

• subscriber/client

Entry point

• gateway or any node

Authoritative decider

• subscription owner/shard leader

Precondition

• no conflicting active subscription edge or expected version matches

Transition

• create or activate SubscriptionEdge

Response

• {subscription_version}

3. Unsubscribe #

API

• Unsubscribe(subscriber_id, topic_id, expected_version?)

Initiator

• subscriber/client

Entry point

• gateway or any node

Authoritative decider

• subscription owner/shard leader

Precondition

• current subscription is active and version matches if supplied

Transition

• mark SubscriptionEdge inactive

Response

• {subscription_version}

4. Fanout delivery creation #

API

• internal fanout worker flow

Initiator

• system/shard leader

Entry point

• shard leader

Authoritative decider

• shard leader

Precondition

• committed publication exists
• authoritative active subscription set available

Transition

• create DeliveryRecord(event_id, subscriber_id) for each matching subscriber
• set initial delivery state

Response

• internal success

Failure cases

• duplicate fanout work deduped by (event_id, subscriber_id)

5. Ack delivery #

API

• AckDelivery(event_id, subscriber_id, attempt_token)

Initiator

• subscriber/client

Entry point

• delivery endpoint / any node

Authoritative decider

• delivery-record owner/shard leader

Precondition

• delivery record is in ackable state
• current attempt token matches

Transition

• mark delivery terminal/acked

Response

• success or stale-token failure

6. Retry delivery #

API

• internal retry scheduler flow

Initiator

• system

Entry point

• shard leader

Authoritative decider

• shard leader

Precondition

• delivery record is retryable
• current state/version allows retry

Transition

• advance delivery attempt
• assign new attempt token
• reschedule actual delivery

Response

• internal success

Entry Point vs Decider vs Responder #

Concrete HLD #

Main components:

• gateway/router
  • resolves topic shard and forwards publication/subscription requests
• shard leader
  • authoritative owner for publication fanout and delivery lifecycle in that shard
• subscription index
  • authoritative subscription edges keyed by topic/subscriber scope
• delivery-state store
  • per-event, per-subscriber delivery lifecycle
• retry scheduler
  • shard-local retry timing and backoff
• metadata/control service
  • tracks shard ownership and routing

Concrete Technology Realizations #

Stronger infra-native answer #

• Go or Java pub/sub service
• RocksDB or append-only segment files for publications and delivery state
• Raft or leader-follower replication per shard
• etcd or internal metadata quorum for shard ownership/routing

Short interview version #

“I’d build the fanout pub/sub platform as a sharded service. Publication is immutable, subscription edges are authoritative relationship state, and the shard leader expands each publication into per-subscriber delivery records. Ack and retry are guarded transitions on those delivery records, fenced by attempt tokens, and hot topics are handled by sharding fanout and keeping status/metrics as derived views.”