Architecture

Continuum is a replicated Automerge document, read and written by path: every node holds the same document, exposes it over an HTTP API, and syncs it with its peers over one or more transports (Mycelium, libp2p, Tor; transports.md). There is no leader and no central service; nodes find each other through seeds and a member registry kept in the document itself. Everything the swarm agrees on, it agrees on through that document.

This page is the map: what the parts are, how they fit, and where the complexity sits. The details live in the other pages (lifecycle.md, operations.md, peers.md, transports.md, remakes.md, api.md, configuration.md, measurements.md, dashboard.md, trust.md, glossary.md) and in the module docs at the top of each source file.

Layers

 operators, applications
          │ HTTP (basic auth)
 ┌────────▼────────────────────────────────────────────────────────────────┐
 │ api/        /store, /stream, /peers, /admin/*, /debug/*                 │
 ├─────────────────────────────────────────────────────────────────────────┤
 │ state/      AppState: the replica, lifecycle, peer states, timings,     │
 │             shared tables; the Host impl for remakes                    │
 ├──────────────────────────────┬──────────────────────────────────────────┤
 │ swarm/                       │ remake/                                  │
 │ peers, sync, relay, topology,│ epochs: proposal, votes, decision,       │
 │ members, blobs, catch-up,    │ snapshot, switch, rejoin, quarantine     │
 │ stability, bootstrap, files  │ (reaches the node only through `Host`)   │
 ├──────────────────────────────┴──────────────────────────────────────────┤
 │ document/   Replica (document + epoch), the store view, the document    │
 │             file                                                        │
 ├─────────────────────────────────────────────────────────────────────────┤
 │ transport/  message types and wire format, traffic counting; the        │
 │             identity layer (node IDs) over the paths layer (addresses,  │
 │             PING/PONG, the debug firewall) over the transports:         │
 │             Mycelium (REST client, daemon), libp2p, Tor                 │
 └─────────────────────────────────────────────────────────────────────────┘
          Mycelium daemon · libp2p peers and relays · the Tor network

Dependencies point downwards, with two deliberate exceptions: state/ is the hub everything else reaches (it holds the shared state and is reached by most modules), and remake/ never touches AppState, peers or sync directly; it sees the node through the Host trait (remake/mod.rs), which AppState implements. That boundary is what keeps the epoch machinery testable with plain documents.

The document

One Automerge document per node (document::replica::Replica: the document, its epoch, cached heads). Automerge is used directly only in document and in what remakes do to documents (remake/: snapshot, delta, catchup, kind, quarantine); the rest of the node names its types as document re-exports them (Document, ChangeHash, Change, ActorId) and works through document's functions. Two views of it:

The store (document::store): JSON over slash-separated paths, with Automerge's concurrent values merged into one view (maps union, scalars newest-wins) and reported on request (?conflicts=true). The HTTP API and the remake kinds work through it; nothing else reads Automerge objects directly. Within a list, a path segment is an index (servers/0/name); a negative one counts from the end. A key with / in it, or an empty one, can be written inside a value but not named in a path: it is read and written through its parent.
Protocol data under _continuum: the swarm ID, the member registry, forget records, remake records, the votes register and epoch metadata. The nodes coordinate through these keys and nothing else: there is no control channel besides the document and the handshake. --protocol-writers may edit them for repairs; the swarm ID and epoch metadata never.

Epochs. The document's history can't be dropped in place, so a remake moves the whole swarm to a new document (same content, no history; or new content), the next epoch. Nodes on different epochs don't sync. The epoch is {generation, id}; the ID is the hash of the new document's first change, which every node builds itself, byte for byte the same (remake/snapshot.rs). That determinism is what lets a node verify it switched to the right document without trusting anyone.

Modules

transport/ — mod.rs defines the message types (HELLO, HELLO_REPLY, sync envelope, STATUS/STATUS_REPLY, PING/PONG, blob messages, LINKS/LINKS_REPLY) and the Transport trait; wire.rs is the wire format (a version byte, bumped with every change to the format, and a kind byte, then the message in MessagePack, derived from the types in mod.rs with codecs for hashes and IDs); mycelium/ (with the mycelium feature) is the REST client (polling inbox, outbound POST, one topic, continuum, for every message), with mycelium/daemon.rs running the daemon (an external one by its URL, or one it starts and supervises); libp2p/ (with the libp2p feature) is libp2p as a transport, direct connections with one message per stream, relays and hole punching (driver.rs, behaviour.rs), the addresses known per peer (book.rs), and which of them are ignored or unroutable (ignore.rs, routes.rs); tor.rs (with the tor feature) is Tor as a transport, an onion service per node and one connection per pair, authenticated by the challenge handshake of challenge.rs; paths.rs makes one address space (<scheme>:<target>) of the node's transports (one or more), routing each send to the transport of its address, measuring the paths to peers on several (PING, PONG) and sending over the best, the next when one fails; identity.rs is the identity layer above it, so the node deals in node IDs only (WHO and ME prove which node is at an address; peers.md); traffic.rs counts messages and bytes per peer; stats.rs the transports' own figures; firewall.rs is the debug firewall (/debug/firewall), which the paths layer applies to everything, by the node at each address; sim.rs (tests only) is a simulated network: many nodes in one process, with per-link delay, loss and duplication from a seed, nodes taken down and brought up, and messages injected as if a node had sent them. It sends the wire encoding, so tests exercise the codec, or a transport's envelope around it (development.md).

simtest/ (tests only) — whole nodes on the simulated network, built as main builds them, with a 250 ms keepalive and a 1 s sync reply timeout: joining and converging, dormancy and return, late messages, a stalled catch-up source; and an ignored 25-node soak test with a propagation bound (cargo test soak -- --ignored). How to build and test quickly: development.md.

swarm/ — the live protocol between nodes:

Module	Does	Documented in
`bootstrap`	genesis, joining through seeds (forever), starting on a document	`lifecycle.md`
`peer/`	handshake, keepalives and healing, peers' epochs, admission with a swarm key, forget records, inbound dispatch, the peer worker (a file each)	`peers.md`
`sync`	the sync worker, Automerge sync messages with retries and reply deadlines	`peers.md`
`changes`	the change publisher: the store paths each change touched (`change_events`)	module docs
`document_file`	the worker keeping the document file (`document::file`) up to date	module docs
`source`	the one peer a catching-up node syncs with, and when to give it up	`peers.md`
`relay`	announcing heads to neighbours and pulling what's missing (partial mesh)	module docs
`links`	answering LINKS with this node's links, for the dashboard's map	`dashboard.md`
`topology/`	which members are neighbours: ring plus low-score links, stable members preferred	module docs
`members`	the member registry: aliases, alive stamps, tiers, the electorate	module docs
`stability`	a node's own connectivity record and its stable/transient tier	module docs
`auth`	the pre-shared swarm key and handshake tags	`trust.md`
`blob`	chunked content-addressed transfers from several peers	module docs
`rtt`, `throughput`	round-trip times (of the paths, from HELLOs until measured) and sync throughput, for the node's own choices	`measurements.md`
`skew`	this node's clock against its peers' (from their PONGs), and a warning while it is off	`measurements.md`
`pair`	which side of a pair leads (sends the first keepalive)	module docs
`stats`	node-wide figures for `/admin/stats`	`measurements.md`
`state_file`	what a node remembers across restarts	`lifecycle.md`

remake/ — epochs and the move between them (remakes.md for operators):

Module	Does
`records`	remake records and the votes register in the document
`api`	`/admin/remakes`: proposing, reports, aborting, superseding
`proposal`	the checks before a proposal, superseding, the electorate and quorum
`vote`	who votes for what, tallies, the one-vote-per-epoch rule
`coordinator`	counts votes, writes the decision, builds the snapshot on commit
`kind`	`compact`, `replace`, `import`: how the new document is built and where later writes go
`snapshot`	the deterministic build and its verification; epoch metadata
`switch`	the remake worker: switching this node to a committed epoch, replaying writes since the base
`catchup`	rejoining a newer epoch from peers (download, verify, replay own writes)
`delta`	replaying changes between two sets of heads into another document
`quarantine`	writes a remake had no place for
`content`	the new document of a `replace`/`import`, fetched by voters
`local`	in-memory per-node remake state: prebuilt snapshots, archive, own actors

document/ — replica.rs (the document with its epoch), store.rs (the JSON view), file.rs (the document file: full save plus appended increments), changes.rs (changes as bytes: one by its hash, and deltas between heads), sync.rs (one sync with a peer, SyncSession: Automerge's sync state with our workarounds for unreliable delivery), meta.rs (the protocol's paths under _continuum: the only place their names are spelled).

api/ — axum handlers: store (/store, /stream), peers, stats/ (JSON and Prometheus), document (backup export), log (filter at runtime), config (/admin/config), health (/health, /ready), debug, auth (htpasswd with Argon2), dashboard/ (with the dashboard feature: the page and its events; dashboard.md), and mod.rs (router, protocol-path rules). The remake routes come from remake/api.rs. docs/openapi.yaml describes it all; nodes serve it at /openapi.yaml, with a Swagger UI at /swagger-ui.

Top level — main.rs reads the configuration, picks the starting document and runs the shutdown sequence; node.rs builds a node's state and starts its workers (also used by the simulator); config/ loads options from defaults, files and arguments (configuration.md); force.rs is the one format for overriding checks; files.rs reads and writes the node's own files and counts them; clock.rs is the node's wall clock (every Unix timestamp it stamps or judges by goes through it, so tests can skew one node's); locks.rs takes std locks without poisoning (lock_or_recover and the like); backoff.rs, logging.rs; testrand.rs (tests only) is a seeded generator and a runner for property tests over many seeds (topology, heads summaries, replay).

Runtime

Tasks. Started by node.rs, running for the life of the process:

the consumer: every transport's inbound messages, through the paths and the identity layer (Mycelium polls its inbox, two polls at once so HTTP round trips overlap; libp2p and Tor read their connections), handed one at a time to peer::handle_message;
the sync lanes: four tasks that apply inbound sync messages, each sender's on one lane in arrival order, so a slow apply doesn't hold up the consumer (peer::lanes);
the sync worker: one queue of dirty peers, coalesced, one Automerge sync message generated per peer per pass and sent by a task of its own, so a slow send holds up no other peer (swarm::sync);
the peer worker: when the earliest is due, keepalives, dormancy, HELLO retries, overdue sync replies; every 5 s (and at once when the node becomes ready), forget records (also when they changed), registry renewal, stability, measurements, the state file (swarm::peer::run_peer_worker);
the topology worker: recomputes the neighbours when the member list or a peer's reachability changes (swarm::topology::run_worker);
the remake worker: every second, on record changes and when a vote is due, voting, switching, rejoining (remake::switch::run_worker);
the change publisher: diffs the document after each change for /stream;
the document file writer, if configured;
bootstrap, until the node is ready, then the HTTP server.

Short-lived tasks: a coordinator per remake this node proposed, blob serving and fetching, the snapshot download of a rejoin, announcement rounds.

Locks. The document is an RwLock<Replica>; what changes it (writes, applying sync messages and fetched changes, the epoch switch) takes the write lock, and what only reads it (generating sync messages, looking changes up, saving) the read lock, both on the blocking pool. Peers are a RwLock<HashMap<_, Arc<Mutex<PeerState>>>>: the map lock is held briefly, each peer's mutex for one operation. Lock order is document, then peer, wherever both are held (sync generation, the epoch switch). Whether a write may go in is checked under the write lock too: a rejoin's fresh document takes none until the rejoin is done (Replica::writable), so a write checked as ready just before the rejoin can't land in it; it goes into the document before the rejoin (StaleView), which is replayed when it ends. A rejoin begins and ends under the write lock, with the lifecycle (taken after the document's) and the stale view. The small shared tables (RTT, throughput, neighbours, stability, catch-up source, remake state) sit behind std mutexes and are never held across an await.

Lifecycle. Joining → Ready, and back to Joining (keeping the swarm ID) while a rejoin runs. Only a ready node registers, votes, serves writes and is followed as a catch-up source. The phases, what each serves and how the files come in: lifecycle.md.

The paths that matter

A write. PUT /store/{path} → AppState::mutate_doc (one Automerge change under the write lock, stamped with the clock) → publish_change (bumps the version, emits DocumentChanged) → the sync worker generates a message per neighbour → each neighbour applies it, relays its new heads to its neighbours, which pull what they lack. In a full mesh (up to k + 1 members) the writer reaches everyone directly.

An inbound message. Consumer → handle_message: drop our own and stale-run replies, admit (swarm key), mark the peer seen, record its epoch (ignoring late ones), then by type: HELLO answered, HELLO_REPLY establishes membership and discovers peers, sync queued on its lane if on our epoch (held back from everyone but the source while catching up), STATUS answered with a STATUS_REPLY and compared for divergence, blobs served or collected, LINKS answered with the node's links if the asker is a member, and again on each change while it subscribes (swarm::links, for the dashboard); PING and PONG (the paths layer answered the PING already) only mark the peer seen, and after a PONG the peer's round-trip time is its path's again.

A remake. Proposal written by the coordinator → every electorate member votes once it holds the content → the coordinator commits on a majority (checking the deadline first, writing the decision atomically with a re-check) → after the switch delay every node builds the snapshot, verifies the epoch ID, replays writes since the base into it and swaps documents → nodes that missed it rejoin: download the snapshot in chunks, sync, replay their own writes. remakes.md has the operator's view, remake/*.rs the rules.

What is kept, and where

Nothing in the document is durable unless --document-file says so: a node that restarts without one starts empty and syncs. The optional files (files.rs):

File	Holds	Written
state file	swarm ID, members to rejoin through	once ready, when the selection changes, at shutdown
document file	the document, this node's actor IDs	appended every few seconds, rewritten on epoch change and growth
backup	`GET /admin/document`, restored with `--recover-from` or imported	by the operator

Everything else (archive of old epochs, quarantine, prebuilt snapshots, measurements, peer states) is in memory and gone at restart.

Design rules

The decisions that recur throughout the code; changing one means revisiting the places that rely on it.

The document is the only shared state. Membership, forgetting, remakes and votes are all keys under _continuum. Nodes never agree on anything out of band.
One writer per key. Each part of a remake record, each registry entry, each vote-register entry has exactly one node that writes it, so concurrent writes never fight. Where two writers are possible anyway (decisions after expiry), the record is marked contested and nobody acts on it alone.
Deterministic builds, verified by hash. A snapshot's bytes depend only on the base document and the proposal, so the epoch ID doubles as proof of the content.
Messages may be late, lost or duplicated. Every message carries epoch and epoch-change count; sync is idempotent and repaired by deadlines; keepalives only inform; a HELLO resets the session on both sides.
Peers on different epochs don't sync. Ever. A node moves epochs by switching (building) or rejoining (downloading), never by merging.
Writes since the base are replayed as state, not history. Per map key, whole lists and texts; by every switching node, and only by their author on a rejoin.
Members are trusted with everything (trust.md). The swarm key is the boundary; nothing inside it is signed.
Conflicts are errors; overrides are named. A contradiction in the configuration or at the API refuses with a message that names the force value that would override it, in one camelCase vocabulary everywhere.
Wire formats grow by appending. New fields default when absent, so a swarm can be upgraded one node at a time (until the first release lifts that constraint).

Where the complexity sits

Roughly by size and by how many rules interact there:

Area	Lines (tests included)	Why it is complex
`remake/`	6,900	Thirteen modules, each simple, but the sequence spans the whole node: proposal, votes, decision, snapshot, switch, rejoin, replay, quarantine, and every failure in between. Most of the project's rules live here.
`swarm/peer/` + `sync.rs` + `source.rs`	2,300	The protocol state machine: handshake, liveness, sync sessions with deadlines, who syncs with whom while catching up. Timing-dependent; tested on the simulated network (`simtest/`).
`state/`	1,600	`AppState` and its write helpers (`mod.rs`), `PeerState` (`peer.rs`), `Timing`, reads (`read.rs`) and the `Host` impl (`host.rs`). A hub, not a layer.
`transport/`	11,500	libp2p (`libp2p/`, 4,400: addresses, relays, hole punching, which paths to keep), the paths and identity layers (2,300: measuring, failing over, proving node IDs), Mycelium and its daemon (`mycelium/`, 1,200), Tor (700), the wire format (`wire.rs`, derived), the simulator. The transports are self-contained behind the `Transport` trait; libp2p's choice of addresses is the subtle part.
`config/`	2,800	Layered sources, importers and placeholders, conflicts, the report. Self-contained.
`document/store.rs`	1,200	Conflict-aware JSON over Automerge. Subtle (merged maps, canonical writes), well covered by the rest relying on it.
`swarm/topology/`, `blob.rs`	1,500	Pure algorithms with tests; the topology hashes each member once, then scores the N² pairs by cheap mixing.
`api/`	3,600	Handlers; the Prometheus rendering is the longest file. The dashboard's page (`dashboard.html`, 2,000 more) is one file of HTML, CSS and script.

The two places where a change needs the most care are peer//sync.rs (every rule about when to send what interacts with every other) and remake/catchup.rs (a rejoin touches the lifecycle, the document swap, blobs, sync and replay at once).