Mobile OS Rollout / Staged Delivery Analysis Note #

This note captures the full step-by-step analysis for a mobile OS rollout / staged delivery service: release metadata, compatibility policy, cohort/stage policy, device eligibility, versioned snapshot propagation, and staged rollout progression.

Step 1 — Normalize #

Assume the baseline prompt is:

• design a mobile OS rollout / staged delivery service
• devices check for updates
• control plane decides which OS version a device is eligible for
• rollout can be staged by cohort, percentage, region, device model, etc.
• rollout can be paused, resumed, or rolled back
• system scales across millions of devices

Notes on normalization:

• device update check is a read path against current effective eligibility state
• release metadata and rollout policy are overwrite-state control-plane objects
• rollout pause/resume/rollback is a lifecycle state transition
• effective eligibility is a recomputed control-plane object
• snapshot propagation is async control-plane dissemination
• install reporting is append-only but secondary to the decision path

This system is a composition of:

• Control Plane + Data Plane
• Time-Bounded Exclusive Allocation flavor for staged eligibility windows

but the dominant architecture is control-plane/data-plane.

Step 2 — Critical Path Selection #

Critical paths:

• P1 evaluate device eligibility
• P2 update release config
• P3 update rollout policy
• P4 transition rollout state
• P5 compute effective eligibility
• P6 propagate config snapshot

Step 3 — Primary State Extraction #

Minimal primary set:

• ReleaseConfig
• RolloutPolicy
• RolloutState
• EligibilityState
• ConfigSnapshot

Important modeling choices:

RolloutState #

This is worth making explicit because rollout lifecycle matters:

• DRAFT
• ACTIVE
• PAUSED
• ROLLED_BACK
• COMPLETED

EligibilityState #

This is the compiled/effective decision state for hot-path checks:

• release compatibility rules
• staged rollout percentage/cohort config
• current rollout state

ConfigSnapshot #

Serving/edge nodes should evaluate against versioned snapshots, not query control plane for every device check.

Step 4 — Hard Invariants #

What matters most:

• devices must not be served releases they are not eligible for
• staged rollout bucketing must be stable for same device and config version
• rollout pause/rollback must take effect within bounded propagation time
• effective eligibility must faithfully represent authoritative release + rollout inputs

Step 5 — Execution Context #

Derived:

• bounded_staleness_allowed = true
• exclusive_claim_required = false
• guarded_by_current_state = true

This implies:

• authoritative control plane
• versioned snapshot propagation
• local eligibility evaluation on hot path

Step 6 — Deterministic Mechanism Selection #

Hot path P1 is a read path.

Why these fit:

• release config and rollout policy are current-state control-plane config
• rollout lifecycle has real state-machine semantics
• effective eligibility is a recomputed current view
• snapshots are versioned overwrite dissemination

Step 7 — Read Model / Source of Truth #

Important point:

For the device-check hot path:

• serving node reads local ConfigSnapshot
• not control-plane source state per request

Step 8 — Failure Handling #

What matters most:

• rollout state transitions must be versioned and legal
• sidecars/serving nodes reject older snapshots
• pause/rollback must propagate within bounded time
• deterministic bucketing must stay stable for the same device and config version

Step 9 — Scale Adjustments #

What scales well:

• hot path is local eligibility evaluation from snapshot
• control plane is narrow and versioned
• propagation is incremental

What fails first:

• snapshot fanout storms during urgent pause/rollback
• large compatibility/policy rule trees
• analytics overload if tied too closely to the decision path

Canonical design conclusion:

• archetype: Control Plane + Data Plane
• primary truth:
  • ReleaseConfig
  • RolloutPolicy
  • RolloutState
  • EligibilityState
  • ConfigSnapshot
• hot path:
  • local snapshot read + deterministic eligibility/bucketing
• control plane:
  • authoritative release config + rollout config + lifecycle + compiled eligibility state + snapshot publication

Concrete Substrate #

• control plane in Go/Java
• authoritative rollout/config store in etcd, Postgres, or another strongly consistent config DB
• config distribution via watch streams / push channels / CDN-backed snapshot pull
• local evaluators at edge nodes, update-check servers, or device-facing services
• optional install-event pipeline via Kafka/PubSub/ClickHouse path

Operation Layer #

1. CheckForUpdate(device_context, current_version, device_key)

• entry point: edge/update-check server
• authoritative decider: local ConfigSnapshot
• transition: none on source truth
• response: {eligible, target_release, config_version, reason}

2. PutReleaseConfig(release_id, config, expected_version?)

• entry point: control-plane API
• authoritative decider: release store owner
• transition: overwrite ReleaseConfig

3. PutRolloutPolicy(scope, config, expected_version?)

• entry point: control-plane API
• authoritative decider: policy store owner
• transition: overwrite RolloutPolicy

4. TransitionRolloutState(rollout_id, new_state, expected_version?)

• entry point: control-plane API
• authoritative decider: rollout-state owner
• transition: guarded update to RolloutState

5. internal recompute

• recompute EligibilityState from release config + rollout policy + rollout state

6. snapshot propagation

• publish latest ConfigSnapshot(version) to serving edges/update-check nodes

7. RecordInstallEvent(device_id, release_id, status, request_id?)

• entry point: async event ingestion endpoint
• authoritative decider: analytics/event pipeline
• transition: append InstallEvent

Entry Point vs Decider vs Responder #

Concrete HLD #

Main components:

• control-plane API
• release + rollout state store
• effective-eligibility compiler/recompute worker
• snapshot distribution layer
• edge/update-check evaluators
• optional install analytics pipeline

Short interview version #

“I’d design the mobile OS rollout system as a control-plane/data-plane service. Control plane stores release metadata, staged rollout policy, and rollout lifecycle state, then compiles them into an effective eligibility snapshot. Devices don’t query control plane deeply on every check; update-check servers evaluate locally from versioned snapshots and use deterministic bucketing so the same device stays in the same treatment for a given config version. Pause, resume, and rollback are explicit lifecycle transitions, and the main correctness boundary is bounded-stale snapshot propagation to the serving edge.”