Random Tech Excursions

Mon, 01 Jan 0001 00:00:00 +0000

AGENTS.md #

Operating contract for agents using this corpus (25 files: 24 canonical blocks + _index.md). Read this file completely before touching the corpus.

What this corpus is #

A closed system of distributed-systems design knowledge:

24 blocks, each owning ONE set of mechanisms
1 capstone (_index.md) owning the meta-layer:
 - 3 enemies (the primitive failure classes)
 - 23 stars (* = per-block technical bottlenecks)
 - 1 ^o (Termination: no recipe exists — FLP)
 - 9 strength ladders under one gradient (hope → proof)
 - the gap/overlap dial (8 sites)
 - the topological design order (11 steps)

Every mechanism is defined in exactly ONE file. Cross-references are written as → file.md or (→ file.md axis N). An arrow means: do not restate; load and cite.

Mon, 01 Jan 0001 00:00:00 +0000

Flow Control / Backpressure #

how does the system prevent demand from overwhelming capacity?

flow control = regulate how fast work enters or moves
backpressure = propagate downstream saturation upstream

Role in the catalog: tier-two protocol block — the dynamics layer. queue.md and scheduler.md own the statics (structures, single-gate decisions); this file owns the load-control loop across a chain of them.

Mon, 01 Jan 0001 00:00:00 +0000

Cache #

cache = derived local copy of truth, held under a freshness contract

It looks like a performance trick. It is actually the deliberate manufacture of staleness, sold for latency:

every other block fights the gap between recorded claim and moving world.
a cache CREATES that gap on purpose and manages the proceeds.

Role in the catalog: the convergence block. Cache is where four other blocks’ machinery meets in one artifact — staleness ( scheduler.md), revocation ( policy.md), the commit point ( queue.md*), and tenant scope (boundary.md). “Cache invalidation and naming things” are not new problems; they are revocation and key-completeness, respectively.

Mon, 01 Jan 0001 00:00:00 +0000

Capacity / Economics #

capacity = how much useful work resources can support
economics = how scarce resources are allocated, paid for, and traded off

The block’s founding claim:

many technically valid systems are still bad designs —
this block is where that judgment is rendered.

Role in the catalog: the evaluation layer. Not a component (like queue, scheduler) and not a protocol (like flow control, recovery) — the layer that prices the others. Scheduler allocates the ledger; backpressure defends useful work against dead work; boundary’s economic boundary says who pays; cache trades one ledger column (compute) for another (memory + staleness risk). This is where VSA stops being a lens and becomes the content.

Mon, 01 Jan 0001 00:00:00 +0000

Checkpoint / Replay #

checkpoint = compact record of progress/state
replay = reapply history after the checkpoint

The block’s founding claim:

I do not need to remember everything as current state,
as long as I can recover current state
from a checkpoint plus surviving history.

Role in the catalog: the recovery-side elaboration of state_machine.md axis 2 (persistence & reconstruction). That axis says whether a machine can be rebuilt; this file owns how, at what cost, bound by what coordinate.

Mon, 01 Jan 0001 00:00:00 +0000

Garbage Collection / Compaction #

The block’s own memory trick, promoted from appendix to thesis — it IS the top-level cleave:

GC asks: what PROVES data is no longer needed?
compaction asks: what PRESERVES MEANING while changing representation?

GC removes. Compaction rewrites.

Two operations, two axes, one shared spine and one shared bottleneck.

Role in the catalog: tier-two protocol block — the maintenance protocol every storage component conjugates. Structurally, checkpoint_replay.md’s dark twin: recovery preserves history, this block destroys it, and the pin registry (below) is the treaty line between them.

Mon, 01 Jan 0001 00:00:00 +0000

Identity / Naming #

identity = what something is
name = how we refer to it
binding = how a name resolves to a thing

Role in the catalog: the substrate of substrates — tier zero. Every coordinate, token, key, and ID the catalog has minted is an instance of this block:

cache keys → cache.md axis 4
idempotency keys → retry_idempotency axis 1
partition keys → shard.md axis 1
fencing epochs → lease_fencing axis 4
binding coordinates → checkpoint_replay*
correlation IDs → telemetry's join key
digests → index_structures.md, versioning.md
principals → policy.md axis 5

policy.md said “you cannot authorize what you cannot name.” The generalization is this block’s charter:

Mon, 01 Jan 0001 00:00:00 +0000

Index #

index = auxiliary structure that maps a question to candidate data

It answers:

where should I look?

Role in the catalog: a component block with a confession — an index is a cache: a materialized derived structure keyed for lookup. It imports cache.md’s freshness ladder wholesale (“index-source divergence” IS staleness), checkpoint_replay.md’s rebuild path (derived state must be reconstructible from source), and garbage_collection.md’s publication protocol (segment generations). What it owns natively is one asymmetry and one bottleneck — both stated below.

Mon, 01 Jan 0001 00:00:00 +0000

Lease / Fencing #

lease = time-bounded authority "who may act, for now?"
fencing = proof that stale authority
 can no longer mutate "how is the old owner stopped?"

Role in the catalog: the keystone. This is the ownership protocol that nine files have been importing — queue’s claim, scheduler’s binding, state machine’s guards, checkpoint’s generation stamp, policy’s short expiry, cache’s coherence rung, GC’s epoch fence, capacity’s reservation, boundary’s crossing credential. One mechanism, defined here once; every consumer thins to a reference. In catalog-VSA terms this block is Lease/Fence* — essential to nine consumers, one known recipe (monotonic token, checked downstream), and until this file, owned nowhere.

Mon, 01 Jan 0001 00:00:00 +0000

Log #

log = append-only sequence of records

It answers:

what happened, in what order, and from where can I replay?

Role in the catalog: the sixth component block — the storage-side substrate, and state_machine.md’s twin: a replicated state machine is literally this block composed with that one (log + deterministic apply).

The queue.md seam, declared up front. queue.md’s axis 1 cleaved remove-on-ack from retained-log; this block IS the retained-log arm, seen from inside. Ownership split:

Mon, 01 Jan 0001 00:00:00 +0000

Materialized View / Projection #

projection = derived state computed from source state/events
materialized view = persisted projection optimized for reads

It answers:

what should we precompute so reads do not redo expensive work?

Role in the catalog: the family portrait — a Combination block. Cache, index, and materialized view are three siblings of one species:

derived, keyed, rebuildable copies of truth

This block creates no new primitive; it names the composite and states the family axes once. Roughly 70% of the classic “projection types” are already owned:

Mon, 01 Jan 0001 00:00:00 +0000

Policy / Capability #

who may do what, to which resource, under which context?

policy = rule for deciding authority
capability = credential that carries authority
enforcement = the place where the decision is applied

Role in the catalog: this block is boundary.md’s crossing discipline, promoted to its own module. boundary.md owns where the lines are; this file owns what happens at the gate.

Mon, 01 Jan 0001 00:00:00 +0000

Queue #

queue = policy-governed waiting set

A queue is the statement:

work exists now,
execution happens later,
under some discipline.

It is where time, work, ownership, fairness, and failure meet.

Design Axes (the core module) #

A “queue type” is not a primitive. It is a point in a five-axis parameter space. Design by choosing a value on each axis; recognize systems by reading their vector.

Mon, 01 Jan 0001 00:00:00 +0000

Consensus / Quorum #

consensus = agree on one value or one ordered sequence of values
quorum = enough participants to make a decision safe

Consensus answers: what is the authoritative decision? Quorum answers: which subset is enough to prove it?

Role in the catalog: the arbiter — the root block. Every other block manages the staleness enemy: bounds it, fences it, names its width. This block is the only one that manufactures facts immune to it — a committed entry is agreed forever; the commit index is the one coordinate in the catalog that never lies. The catalog’s strongest guarantees are purchased here and exported:

Mon, 01 Jan 0001 00:00:00 +0000

Replication #

replication = maintain multiple copies of state

It answers:

where else does this state exist, and how do copies converge?

Role in the catalog: the topology layer — and the block whose rewrite is mostly treaty-writing, because it arrives after quorum.md, log.md, and lease_fencing.md claimed the strong machinery. The treaty:

quorum.md owns the commit proof (sync/quorum replication, SMR,
 leaderless R/W — its two arms)
log.md owns the shipped structure (log shipping, ordering, the ladder)
lease_fencing.md owns authority transfer (failover fencing, split brain)
replication.md owns the TOPOLOGY (the geometry of copies, placement,
 and everything that happens where
 the strong machinery ISN'T running:
 async tails, accepted conflict,
 repair, promotion)

The native territory is precisely the part of the copy-universe that consensus does not govern.

Mon, 01 Jan 0001 00:00:00 +0000

Retry / Idempotency #

The block’s premise, and the catalog’s most annoying fact:

a timeout does not tell you whether the operation happened.

That is state_machine.md’s ignorance* verbatim — and this block is what you DO about it:

retry = repeat an operation after an uncertain outcome
idempotency = make the repetition safe

Role in the catalog: the second reunion block — the duplication protocol. lease_fencing.md gathered nine files’ ownership fragments; this file gathers the commit point* — stated in queue.md, cited by six files since — and gives it its recipe book. queue.md owns the star’s statement; this file owns its practice. The homecomings:

Mon, 01 Jan 0001 00:00:00 +0000

Routing / Load Balancing #

routing = choose path or destination
load balancing = choose among equivalent destinations
 under capacity/health policy

It answers:

where should this request/packet/message/query go NOW?

Role in the catalog: the traffic face — the block with the highest import ratio, because nearly every classic “type” is an already- adjudicated concern arriving in a routing costume:

Mon, 01 Jan 0001 00:00:00 +0000

Scheduler #

scheduler = policy engine that turns scarce capacity into ordered, placed, admitted work

A scheduler is more than a queue:

queue = policy-governed waiting set
scheduler = queue + capacity model + decision discipline + binding

Central tension (the objective space):

fairness, latency, throughput, locality, utilization, isolation
cannot all be maximized at once.
Every discipline is a weighting of this vector.
Every failure mode is an under-weighted objective presenting its bill.

Design Axes (the core module) #

A “scheduler type” is a point in a five-axis space. Design by choosing per axis; recognize systems by reading their vector.

Mon, 01 Jan 0001 00:00:00 +0000

Partition / Shard #

partition = subset of a larger space
shard = independently placed/owned partition

It answers:

which slice owns this key/work/query/state?

Role in the catalog: the operator block. Every block so far has been a component or a protocol; this one is a modular operator wearing an infrastructure costume — partitioning is Splitting, applied to state. So the axes are the standard Splitting questions: what is the cleaving key, which invariants stay local, what crosses the cut, and how do the modules move. And the classic confusions are all one category error:

Mon, 01 Jan 0001 00:00:00 +0000

Snapshot / Read View #

snapshot = a coherent version of state
read view = the rules deciding what a reader may see

It answers:

which state is visible to THIS read, while everything moves?

Role in the catalog: the visibility protocol — the reader’s side of the story. checkpoint_replay.md owns how state is captured; GC owns when it dies; this block owns what a reader is ENTITLED TO SEE in between. Every other file’s pin registry, publication protocol, and freshness ladder silently assumed a reader holding a coherent view; this is the block that defines one. It closes a symmetry the catalog has had half of all along: checkpoint_replay produces the coordinate; this block consumes it.

Mon, 01 Jan 0001 00:00:00 +0000

State Machine #

state machine = state + allowed transitions + invariants

It answers:

what states can this thing be in,
what events/actions move it,
and what must never be violated?

Role in the catalog: substrate, not peer. Queues, schedulers, boundaries, transactions, controllers, and protocols are built out of state machines. This block owns the formalism; the component blocks own their instantiations.

Mon, 01 Jan 0001 00:00:00 +0000

Telemetry / Feedback #

telemetry = structured observations about system behavior
feedback = decisions and actions driven by those observations

It answers:

what is happening, why, and what should change?

Role in the catalog: the evaluation layer’s second member — predicted twice before it arrived. capacity.md forecast it (“SLOs are the availability-denominated sibling of unit economics”); log.md evicted trace logs to it; index_structures.md forecast that “the metric you didn’t emit” would be silent omission’s next member. All three debts land here:

Mon, 01 Jan 0001 00:00:00 +0000

Transaction #

transaction = a set of operations treated as one logical change

It answers:

what must happen together, or not at all?

Role in the catalog: the consistency boundary, elaborated — and the atomicity-domain factory. boundary.md listed the consistency boundary as type #10 and never developed it; this file is that development. And retry_idempotency.md CONSUMES atomicity domains (“put the marker inside one”); this block is where domains are MANUFACTURED. The homecomings are heavy:

Mon, 01 Jan 0001 00:00:00 +0000

Versioning / Compatibility #

versioning = naming which shape/semantics something has
compatibility = the ability of different versions to coexist safely

It answers:

what must keep working while everything changes?

Role in the catalog: the evolution protocol — the time-axis dual. Every previous block manages coexistence across SPACE (replicas, shards, caches: copies of state at one moment); this block manages coexistence across TIME: old and new versions of code, schema, protocol, and behavior sharing one running system during the only interval that matters — the transition. At scale, the mixed fleet is the permanent condition, not the transition.