diff options
Diffstat (limited to 'docs/architecture.md')
| -rw-r--r-- | docs/architecture.md | 306 |
1 files changed, 306 insertions, 0 deletions
diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..e0e2c94 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,306 @@ +# Architecture + +Continuum is a replicated Automerge document, read and written by path: every node +holds the same document, exposes it over an HTTP API, and syncs it with its peers over one or more +transports (Mycelium, libp2p, Tor; [`transports.md`](transports.md)). There +is no leader and no central service; nodes find each other through seeds and a member +registry kept in the document itself. Everything the swarm agrees on, it agrees on +through that document. + +This page is the map: what the parts are, how they fit, and where the complexity sits. +The details live in the other pages ([`lifecycle.md`](lifecycle.md), [`operations.md`](operations.md), [`peers.md`](peers.md), [`transports.md`](transports.md), +[`remakes.md`](remakes.md), [`api.md`](api.md), [`configuration.md`](configuration.md), [`measurements.md`](measurements.md), [`dashboard.md`](dashboard.md), [`trust.md`](trust.md), +[`glossary.md`](glossary.md)) and +in the module docs at the top of each source file. + +## Layers + +``` + operators, applications + │ HTTP (basic auth) + ┌────────▼────────────────────────────────────────────────────────────────┐ + │ api/ /store, /stream, /peers, /admin/*, /debug/* │ + ├─────────────────────────────────────────────────────────────────────────┤ + │ state/ AppState: the replica, lifecycle, peer states, timings, │ + │ shared tables; the Host impl for remakes │ + ├──────────────────────────────┬──────────────────────────────────────────┤ + │ swarm/ │ remake/ │ + │ peers, sync, relay, topology,│ epochs: proposal, votes, decision, │ + │ members, blobs, catch-up, │ snapshot, switch, rejoin, quarantine │ + │ stability, bootstrap, files │ (reaches the node only through `Host`) │ + ├──────────────────────────────┴──────────────────────────────────────────┤ + │ document/ Replica (document + epoch), the store view, the document │ + │ file │ + ├─────────────────────────────────────────────────────────────────────────┤ + │ transport/ message types and wire format, traffic counting; the │ + │ identity layer (node IDs) over the paths layer (addresses, │ + │ PING/PONG, the debug firewall) over the transports: │ + │ Mycelium (REST client, daemon), libp2p, Tor │ + └─────────────────────────────────────────────────────────────────────────┘ + Mycelium daemon · libp2p peers and relays · the Tor network +``` + +Dependencies point downwards, with two deliberate exceptions: [`state/`](../src/state/) is the hub +everything else reaches (it holds the shared state and is reached by most modules), and +[`remake/`](../src/remake/) never touches `AppState`, peers or sync directly; it sees the node through +the `Host` trait ([`remake/mod.rs`](../src/remake/mod.rs)), which `AppState` implements. That boundary is what +keeps the epoch machinery testable with plain documents. + +## The document + +One Automerge document per node (`document::replica::Replica`: the document, its epoch, +cached heads). Automerge is used directly only in `document` and in what remakes do to +documents ([`remake/`](../src/remake/): `snapshot`, `delta`, `catchup`, `kind`, `quarantine`); the rest of +the node names its types as `document` re-exports them (`Document`, `ChangeHash`, +`Change`, `ActorId`) and works through `document`'s functions. Two views of it: + +* **The store** (`document::store`): JSON over slash-separated paths, with Automerge's + concurrent values merged into one view (maps union, scalars newest-wins) and + reported on request (`?conflicts=true`). The HTTP API and the remake kinds work + through it; nothing else reads Automerge objects directly. Within a list, a path + segment is an index (`servers/0/name`); a negative one counts from the end. A key with `/` in it, or + an empty one, can be written inside a value but not named in a path: it is read and + written through its parent. +* **Protocol data** under `_continuum`: the swarm ID, the member registry, forget + records, remake records, the votes register and epoch metadata. The nodes coordinate + through these keys and nothing else: there is no control channel besides the + document and the handshake. `--protocol-writers` may edit them for repairs; the + swarm ID and epoch metadata never. + +**Epochs.** The document's history can't be dropped in place, so a *remake* moves the +whole swarm to a new document (same content, no history; or new content), the next +epoch. Nodes on different epochs don't sync. The epoch is `{generation, id}`; the ID +is the hash of the new document's first change, which every node builds itself, byte +for byte the same ([`remake/snapshot.rs`](../src/remake/snapshot.rs)). That determinism is what lets a node verify +it switched to the right document without trusting anyone. + +## Modules + +**[`transport/`](../src/transport/)** — [`mod.rs`](../src/transport/mod.rs) defines the message types (HELLO, HELLO_REPLY, sync +envelope, STATUS/STATUS_REPLY, PING/PONG, blob messages, LINKS/LINKS_REPLY) and the `Transport` trait; +[`wire.rs`](../src/transport/wire.rs) is the wire format (a version byte, bumped with every change to the format, +and a kind byte, then the message in MessagePack, derived from the types in [`mod.rs`](../src/transport/mod.rs) +with codecs for hashes and IDs); [`mycelium/`](../src/transport/mycelium/) (with the `mycelium` feature) is the REST +client (polling inbox, outbound POST, one topic, `continuum`, for every message), with +[`mycelium/daemon.rs`](../src/transport/mycelium/daemon.rs) running the daemon (an external one by its URL, or one it starts +and supervises); [`libp2p/`](../src/transport/libp2p/) (with the `libp2p` feature) is libp2p as a transport, +direct connections with one message per stream, relays and hole punching ([`driver.rs`](../src/transport/libp2p/driver.rs), +[`behaviour.rs`](../src/transport/libp2p/behaviour.rs)), the addresses known per peer ([`book.rs`](../src/transport/libp2p/book.rs)), and which of them are +ignored or unroutable ([`ignore.rs`](../src/transport/libp2p/ignore.rs), [`routes.rs`](../src/transport/libp2p/routes.rs)); [`tor.rs`](../src/transport/tor.rs) (with the `tor` feature) is Tor as a +transport, an onion service per node and one connection per pair, authenticated by the +challenge handshake of [`challenge.rs`](../src/transport/challenge.rs); [`paths.rs`](../src/transport/paths.rs) makes one address space +(`<scheme>:<target>`) of the node's transports (one or more), routing each send to the +transport of its address, measuring the paths to peers on several (PING, PONG) and +sending over the best, the next when one fails; [`identity.rs`](../src/transport/identity.rs) is the identity layer above it, so the node +deals in node IDs only (WHO and ME prove which node is at an address; [`peers.md`](peers.md)); [`traffic.rs`](../src/transport/traffic.rs) counts messages and +bytes per peer; [`stats.rs`](../src/transport/stats.rs) the transports' own figures; [`firewall.rs`](../src/transport/firewall.rs) is the debug firewall (`/debug/firewall`), which the paths layer applies to +everything, by the node at each address; [`sim.rs`](../src/transport/sim.rs) (tests only) is a +simulated network: many nodes in one process, with per-link delay, loss and +duplication from a seed, nodes taken down and brought up, and messages injected as if +a node had sent them. It sends the wire encoding, so tests exercise the codec, or a +transport's envelope around it ([`development.md`](development.md)). + +**[`simtest/`](../src/simtest/)** (tests only) — whole nodes on the simulated network, built as `main` +builds them, with a 250 ms keepalive and a 1 s sync reply timeout: joining and +converging, dormancy and return, late messages, a stalled catch-up source; and an +ignored 25-node soak test with a propagation bound (`cargo test soak -- --ignored`). How to build and test quickly: [`development.md`](development.md). + +**[`swarm/`](../src/swarm/)** — the live protocol between nodes: + +| Module | Does | Documented in | +|---|---|---| +| `bootstrap` | genesis, joining through seeds (forever), starting on a document | [`lifecycle.md`](lifecycle.md) | +| [`peer/`](../src/swarm/peer/) | handshake, keepalives and healing, peers' epochs, admission with a swarm key, forget records, inbound dispatch, the peer worker (a file each) | [`peers.md`](peers.md) | +| `sync` | the sync worker, Automerge sync messages with retries and reply deadlines | [`peers.md`](peers.md) | +| `changes` | the change publisher: the store paths each change touched (`change_events`) | module docs | +| `document_file` | the worker keeping the document file (`document::file`) up to date | module docs | +| `source` | the one peer a catching-up node syncs with, and when to give it up | [`peers.md`](peers.md) | +| `relay` | announcing heads to neighbours and pulling what's missing (partial mesh) | module docs | +| `links` | answering LINKS with this node's links, for the dashboard's map | [`dashboard.md`](dashboard.md) | +| [`topology/`](../src/swarm/topology/) | which members are neighbours: ring plus low-score links, stable members preferred | module docs | +| `members` | the member registry: aliases, alive stamps, tiers, the electorate | module docs | +| `stability` | a node's own connectivity record and its stable/transient tier | module docs | +| `auth` | the pre-shared swarm key and handshake tags | [`trust.md`](trust.md) | +| `blob` | chunked content-addressed transfers from several peers | module docs | +| `rtt`, `throughput` | round-trip times (of the paths, from HELLOs until measured) and sync throughput, for the node's own choices | [`measurements.md`](measurements.md) | +| `skew` | this node's clock against its peers' (from their PONGs), and a warning while it is off | [`measurements.md`](measurements.md) | +| `pair` | which side of a pair leads (sends the first keepalive) | module docs | +| `stats` | node-wide figures for `/admin/stats` | [`measurements.md`](measurements.md) | +| `state_file` | what a node remembers across restarts | [`lifecycle.md`](lifecycle.md) | + +**[`remake/`](../src/remake/)** — epochs and the move between them ([`remakes.md`](remakes.md) for operators): + +| Module | Does | +|---|---| +| `records` | remake records and the votes register in the document | +| `api` | `/admin/remakes`: proposing, reports, aborting, superseding | +| `proposal` | the checks before a proposal, superseding, the electorate and quorum | +| `vote` | who votes for what, tallies, the one-vote-per-epoch rule | +| `coordinator` | counts votes, writes the decision, builds the snapshot on commit | +| `kind` | `compact`, `replace`, `import`: how the new document is built and where later writes go | +| `snapshot` | the deterministic build and its verification; epoch metadata | +| `switch` | the remake worker: switching this node to a committed epoch, replaying writes since the base | +| `catchup` | rejoining a newer epoch from peers (download, verify, replay own writes) | +| `delta` | replaying changes between two sets of heads into another document | +| `quarantine` | writes a remake had no place for | +| `content` | the new document of a `replace`/`import`, fetched by voters | +| `local` | in-memory per-node remake state: prebuilt snapshots, archive, own actors | + +**[`document/`](../src/document/)** — [`replica.rs`](../src/document/replica.rs) (the document with its epoch), [`store.rs`](../src/document/store.rs) (the JSON +view), [`file.rs`](../src/document/file.rs) (the document file: full save plus appended increments), [`changes.rs`](../src/document/changes.rs) +(changes as bytes: one by its hash, and deltas between heads), [`sync.rs`](../src/document/sync.rs) (one sync with +a peer, `SyncSession`: Automerge's sync state with our workarounds for unreliable +delivery), [`meta.rs`](../src/document/meta.rs) (the protocol's paths under `_continuum`: the only place their +names are spelled). + +**[`api/`](../src/api/)** — axum handlers: `store` (`/store`, `/stream`), `peers`, [`stats/`](../src/api/stats/) (JSON and +Prometheus), `document` (backup export), `log` (filter at runtime), `config` +(`/admin/config`), `health` (`/health`, `/ready`), `debug`, `auth` (htpasswd with +Argon2), [`dashboard/`](../src/api/dashboard/) (with the `dashboard` feature: the page and its events; +[`dashboard.md`](dashboard.md)), and [`mod.rs`](../src/api/mod.rs) (router, protocol-path rules). The remake routes +come from [`remake/api.rs`](../src/remake/api.rs). [`docs/openapi.yaml`](openapi.yaml) describes it all; nodes serve it at +`/openapi.yaml`, with a Swagger UI at `/swagger-ui`. + +**Top level** — [`main.rs`](../src/main.rs) reads the configuration, picks the starting document and +runs the shutdown sequence; [`node.rs`](../src/node.rs) builds a node's state and starts its workers +(also used by the simulator); +[`config/`](../src/config/) loads options from defaults, files and arguments ([`configuration.md`](configuration.md)); +[`force.rs`](../src/force.rs) is the one format for overriding checks; [`files.rs`](../src/files.rs) reads and writes the +node's own files and counts them; [`clock.rs`](../src/clock.rs) is the node's wall clock (every Unix +timestamp it stamps or judges by goes through it, so tests can skew one node's); +[`locks.rs`](../src/locks.rs) takes std locks without poisoning (`lock_or_recover` and the like); +[`backoff.rs`](../src/backoff.rs), [`logging.rs`](../src/logging.rs); [`testrand.rs`](../src/testrand.rs) (tests only) is a seeded generator and a +runner for property tests over many seeds (topology, heads summaries, replay). + +## Runtime + +**Tasks.** Started by [`node.rs`](../src/node.rs), running for the life of the process: + +* the **consumer**: every transport's inbound messages, through the paths and the + identity layer (Mycelium polls its inbox, two polls at once so HTTP round trips + overlap; libp2p and Tor read their connections), handed one at a time to + `peer::handle_message`; +* the **sync lanes**: four tasks that apply inbound sync messages, each sender's on one + lane in arrival order, so a slow apply doesn't hold up the consumer + (`peer::lanes`); +* the **sync worker**: one queue of dirty peers, coalesced, one Automerge sync + message generated per peer per pass and sent by a task of its own, so a slow send + holds up no other peer (`swarm::sync`); +* the **peer worker**: when the earliest is due, keepalives, dormancy, HELLO retries, + overdue sync replies; every 5 s (and at once when the node becomes ready), forget records + (also when they changed), registry renewal, stability, measurements, the state file + (`swarm::peer::run_peer_worker`); +* the **topology worker**: recomputes the neighbours when the member list or a peer's + reachability changes (`swarm::topology::run_worker`); +* the **remake worker**: every second, on record changes and when a vote is due, + voting, switching, rejoining (`remake::switch::run_worker`); +* the **change publisher**: diffs the document after each change for `/stream`; +* the **document file** writer, if configured; +* **bootstrap**, until the node is ready, then the HTTP server. + +Short-lived tasks: a coordinator per remake this node proposed, blob serving and +fetching, the snapshot download of a rejoin, announcement rounds. + +**Locks.** The document is an `RwLock<Replica>`; what changes it (writes, applying +sync messages and fetched changes, the epoch switch) takes the write lock, and what +only reads it (generating sync messages, looking changes up, saving) the read lock, +both on the blocking pool. Peers are a `RwLock<HashMap<_, Arc<Mutex<PeerState>>>>`: the map lock +is held briefly, each peer's mutex for one operation. Lock order is document, then +peer, wherever both are held (sync generation, the epoch switch). Whether a write may +go in is checked under the write lock too: a rejoin's fresh document takes none until +the rejoin is done (`Replica::writable`), so a write checked as ready just before the +rejoin can't land in it; it goes into the document before the rejoin (`StaleView`), +which is replayed when it ends. A rejoin begins and ends under the write lock, with +the lifecycle (taken after the document's) and the stale view. The small shared +tables (RTT, throughput, neighbours, stability, catch-up source, remake state) sit +behind std mutexes and are never held across an await. + +**Lifecycle.** `Joining` → `Ready`, and back to `Joining` (keeping the swarm ID) while +a rejoin runs. Only a ready node registers, votes, serves writes and is followed as a +catch-up source. The phases, what each serves and how the files come in: [`lifecycle.md`](lifecycle.md). + +## The paths that matter + +**A write.** `PUT /store/{path}` → `AppState::mutate_doc` (one Automerge change under +the write lock, stamped with the clock) → `publish_change` (bumps the version, emits +`DocumentChanged`) → the sync worker generates a message per neighbour → each +neighbour applies it, relays its new heads to *its* neighbours, which pull what they +lack. In a full mesh (up to `k + 1` members) the writer reaches everyone directly. + +**An inbound message.** Consumer → `handle_message`: drop our own and stale-run +replies, admit (swarm key), mark the peer seen, record its epoch (ignoring late ones), +then by type: HELLO answered, HELLO_REPLY establishes membership and discovers peers, +sync queued on its lane if on our epoch (held back from everyone but the source while catching +up), STATUS answered with a STATUS_REPLY and compared for divergence, blobs served or collected, +LINKS answered with the node's links if the asker is a member, and again on each change while it subscribes (`swarm::links`, for the dashboard); +PING and PONG (the paths layer answered the PING already) only mark the peer seen, and +after a PONG the peer's round-trip time is its path's again. + +**A remake.** Proposal written by the coordinator → every electorate member votes once +it holds the content → the coordinator commits on a majority (checking the deadline +first, writing the decision atomically with a re-check) → after the switch delay +every node builds the snapshot, verifies the epoch ID, replays writes since the base +into it and swaps documents → nodes that missed it rejoin: download the snapshot in +chunks, sync, replay their own writes. [`remakes.md`](remakes.md) has the operator's view, +[`remake/*.rs`](../src/remake/) the rules. + +## What is kept, and where + +Nothing in the document is durable unless `--document-file` says so: a node that +restarts without one starts empty and syncs. The optional files ([`files.rs`](../src/files.rs)): + +| File | Holds | Written | +|---|---|---| +| state file | swarm ID, members to rejoin through | once ready, when the selection changes, at shutdown | +| document file | the document, this node's actor IDs | appended every few seconds, rewritten on epoch change and growth | +| backup | `GET /admin/document`, restored with `--recover-from` or imported | by the operator | + +Everything else (archive of old epochs, quarantine, prebuilt snapshots, measurements, +peer states) is in memory and gone at restart. + +## Design rules + +The decisions that recur throughout the code; changing one means revisiting the +places that rely on it. + +* **The document is the only shared state.** Membership, forgetting, remakes and votes + are all keys under `_continuum`. Nodes never agree on anything out of band. +* **One writer per key.** Each part of a remake record, each registry entry, each + vote-register entry has exactly one node that writes it, so concurrent writes never + fight. Where two writers are possible anyway (decisions after expiry), the record + is marked contested and nobody acts on it alone. +* **Deterministic builds, verified by hash.** A snapshot's bytes depend only on the + base document and the proposal, so the epoch ID doubles as proof of the content. +* **Messages may be late, lost or duplicated.** Every message carries epoch and + epoch-change count; sync is idempotent and repaired by deadlines; keepalives only + inform; a HELLO resets the session on both sides. +* **Peers on different epochs don't sync.** Ever. A node moves epochs by switching + (building) or rejoining (downloading), never by merging. +* **Writes since the base are replayed as state, not history.** Per map key, whole + lists and texts; by every switching node, and only by their author on a rejoin. +* **Members are trusted with everything** ([`trust.md`](trust.md)). The swarm key is the + boundary; nothing inside it is signed. +* **Conflicts are errors; overrides are named.** A contradiction in the configuration + or at the API refuses with a message that names the `force` value that would + override it, in one camelCase vocabulary everywhere. +* **Wire formats grow by appending.** New fields default when absent, so a swarm can + be upgraded one node at a time (until the first release lifts that constraint). + +## Where the complexity sits + +Roughly by size and by how many rules interact there: + +| Area | Lines (tests included) | Why it is complex | +|---|---|---| +| [`remake/`](../src/remake/) | 6,900 | Thirteen modules, each simple, but the sequence spans the whole node: proposal, votes, decision, snapshot, switch, rejoin, replay, quarantine, and every failure in between. Most of the project's rules live here. | +| [`swarm/peer/`](../src/swarm/peer/) + [`sync.rs`](../src/swarm/sync.rs) + [`source.rs`](../src/swarm/source.rs) | 2,300 | The protocol state machine: handshake, liveness, sync sessions with deadlines, who syncs with whom while catching up. Timing-dependent; tested on the simulated network ([`simtest/`](../src/simtest/)). | +| [`state/`](../src/state/) | 1,600 | `AppState` and its write helpers ([`mod.rs`](../src/state/mod.rs)), `PeerState` ([`peer.rs`](../src/state/peer.rs)), `Timing`, reads ([`read.rs`](../src/state/read.rs)) and the `Host` impl ([`host.rs`](../src/state/host.rs)). A hub, not a layer. | +| [`transport/`](../src/transport/) | 11,500 | libp2p ([`libp2p/`](../src/transport/libp2p/), 4,400: addresses, relays, hole punching, which paths to keep), the paths and identity layers (2,300: measuring, failing over, proving node IDs), Mycelium and its daemon ([`mycelium/`](../src/transport/mycelium/), 1,200), Tor (700), the wire format ([`wire.rs`](../src/transport/wire.rs), derived), the simulator. The transports are self-contained behind the `Transport` trait; libp2p's choice of addresses is the subtle part. | +| [`config/`](../src/config/) | 2,800 | Layered sources, importers and placeholders, conflicts, the report. Self-contained. | +| [`document/store.rs`](../src/document/store.rs) | 1,200 | Conflict-aware JSON over Automerge. Subtle (merged maps, canonical writes), well covered by the rest relying on it. | +| [`swarm/topology/`](../src/swarm/topology/), [`blob.rs`](../src/swarm/blob.rs) | 1,500 | Pure algorithms with tests; the topology hashes each member once, then scores the N² pairs by cheap mixing. | +| [`api/`](../src/api/) | 3,600 | Handlers; the Prometheus rendering is the longest file. The dashboard's page ([`dashboard.html`](../src/api/dashboard/dashboard.html), 2,000 more) is one file of HTML, CSS and script. | + +The two places where a change needs the most care are [`peer/`](../src/swarm/peer/)/[`sync.rs`](../src/swarm/sync.rs) (every +rule about when to send what interacts with every other) and [`remake/catchup.rs`](../src/remake/catchup.rs) +(a rejoin touches the lifecycle, the document swap, blobs, sync and replay at once). |