diff options
Diffstat (limited to 'docs/lifecycle.md')
| -rw-r--r-- | docs/lifecycle.md | 238 |
1 files changed, 238 insertions, 0 deletions
diff --git a/docs/lifecycle.md b/docs/lifecycle.md new file mode 100644 index 0000000..9fe9442 --- /dev/null +++ b/docs/lifecycle.md @@ -0,0 +1,238 @@ +# The life of a node + +A node passes through a few phases between starting and stopping. This page says what +each phase is for, how a node enters and leaves it, and what it does and doesn't do +meanwhile. The protocol between peers is in [`peers.md`](peers.md), remakes in [`remakes.md`](remakes.md); this +page refers to them rather than repeating them. + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Starting + Starting --> Ready: --genesis, --recover-from,<br/>or a document file holding a document + Starting --> Joining: otherwise + Joining --> Ready: swarm ID known, initial sync done,<br/>no rejoin running + Ready --> Rejoining: enough ready members<br/>advertise a newer epoch + Rejoining --> Ready: epoch verified, own writes<br/>replayed or quarantined + Rejoining --> Ready: abandoned - old document restored + Ready --> Stopping: SIGINT, SIGTERM,<br/>or a transport stops + Joining --> Stopping + Stopping --> [*] +``` + +In the code there are two lifecycle states, `Joining` and `Ready` (`NodeLifecycle`); +rejoining is `Joining` with the swarm ID kept, and starting and stopping are what +[`main.rs`](../src/main.rs) does before and after. The phases below are the operator's view. + +| Phase | Reads | Writes | Registers, votes, syncs as a source | Files written | +|---|---|---|---|---| +| Starting | — | — | no | — | +| Joining | `503` (`GET /store` root: the live, partial document) | `503` | no | — | +| Ready | live | yes | yes | state file, document file | +| Rejoining | the pre-rejoin document, with `x-continuum-stale` | into the pre-rejoin document, carried over at the end | no | document file (the pre-rejoin document) | +| Stopping | draining | draining | no | state file, document file | + +`/health`, `/peers`, `/admin/stats`, `/admin/config` and `/admin/log` answer in every +phase, and `/ready` says `200` only when ready (both without authentication); the +remake endpoints and `GET /admin/document` need a ready node. + +## Starting + +What happens before the node talks to anyone, in order: + +1. **Configuration** is loaded from defaults, files and arguments ([`configuration.md`](configuration.md)). + Contradictions are errors; `--print-config` shows the result and exits. +2. **The node key** (`--node-key-file`) and **the swarm key** (`--swarm-key-file`) are + read, so a bad file fails at once. A missing key file is an error, unless + `--node-key-create` asks for a new node key (a new node), or `--swarm-key-create` + for a new swarm key (on the swarm's first node; the others need a copy). +3. **The state file** (`--state-file`) is read: the swarm ID the node belongs to and + members to reach it through. +4. **A document to start on**, if any: the document file (`--document-file`) or a + backup (`--recover-from`). It names the swarm and the epoch the node is on. A new + swarm's document is made here too (`--genesis --remake KIND:PATH`). +5. **The transports** (those enabled), in order: for Mycelium, + the node connects to a running daemon (`--mycelium-rest-url`) or starts and + supervises one; Tor bootstraps arti. A signal stops the node while they start. +6. **The workers** start: message consumer, sync, peers, remakes. +7. **The HTTP API** listens (`--bind-addr`), also while the node joins (the table + above says what it answers then). + +Then the node is either ready at once or joining: + +| Start | Without a remembered swarm | With one (state file) | +|---|---|---| +| `--genesis` | a new swarm, ready at once | that swarm again, empty; nodes of it still running merge with this one | +| document file holding a document | ready at once, on the document's swarm and epoch | the same; the two must name the same swarm | +| `--recover-from` | ready at once, on the backup's swarm and epoch | the same; the two must name the same swarm | +| seeds, or none | joining through the seeds | joining that swarm, through the seeds and the remembered members | + +Conflicts between these inputs are errors unless overridden with `--force` +([`configuration.md`](configuration.md), "Forcing"): `swarmId` (the document names another swarm than the +state file remembers), `badDocumentFile` (the file can't be read: moved aside, start +without it), `replaceDocumentFile` (`--recover-from` while the document file holds a +document: the backup wins). `--genesis` with seeds, with `--recover-from`, or with a +document file holding a document is an error with no override. + +### What the files are for + +* **The state file** holds the swarm ID and the addresses of up to 16 members alive + within the electorate window (stable ones first), and the members' aliases, so the + logs name members by them from the start (perhaps stale; a change is logged once the + member list says so; `--log-raw-ids` leaves IDs as they are). With it, a restarted node joins only its own swarm (its + HELLOs claim the remembered ID) and finds it without configured seeds. Without it, + a node remembers nothing, and `--genesis` on every start makes another new swarm + while the old one's nodes go on without it. The file is written when the node becomes + ready, when the remembered members change (checked every 5 minutes, written at most + hourly), and at shutdown; atomically; an unreadable one is moved aside (`.bad`). +* **The document file** holds the document itself and the actor IDs this node wrote + with, so a restart resumes with the data and a later rejoin still replays the node's + own writes. It is appended to every `--document-append` (default 5s) and + rewritten after an epoch change, a restart, or when the appended part outgrew the + full save (at most every `--document-rewrite`, default 1h); a record cut short + by a crash is dropped when reading. A node starting from it is ready at once; sync + merges what it missed, or it rejoins a newer epoch. +* **A backup** (`GET /admin/document`) is the whole document with its history, naming + its swarm and epoch. `--recover-from` starts a node on it; `--genesis --remake + import:FILE` starts a *new* swarm with its content; an `import` remake brings its + content into a running swarm ([`remakes.md`](remakes.md)). + +### A whole swarm restarting + +If every node restarts without document files, none has any data: they all join, wait +for each other, and none becomes ready. Start one with `--genesis` (with a state file: +the same swarm again, empty) or with `--recover-from` and a backup; the others join +it. Nodes with document files are ready at once and merge with each other. + +## Joining + +The node knows no swarm (or only the remembered one) and has no document. It resolves +its seeds and says HELLO; the first HELLO_REPLY tells it the swarm ID and other peers +to greet. It then catches up: it syncs with one peer only, the *catch-up source*, so +that not every peer sends it the whole document at once; the others wait and are +answered once the initial sync is done ([`peers.md`](peers.md), "Handshake and join", says how +the source is chosen and given up). + +A joining node never gives up: seeds that don't resolve are retried with backoff, +unanswered HELLOs are resent, and every five minutes it logs why it is still joining. +A node without seeds and without `--genesis` has nothing to join and waits (seeds may +be added later). + +If the swarm has moved on to a newer epoch than the node's (a new node starts at +genesis), the node rejoins while still joining: it downloads that epoch's snapshot +first, then catches up. Bootstrap waits for the rejoin, so the node becomes ready only +after the epoch check. + +The node becomes **ready** when it knows the swarm ID, its initial sync is done (a +ready peer with a non-empty document has nothing left that the node lacks) and no +rejoin is running. + +## Ready + +The normal phase. The node: + +* serves reads and writes, and syncs with its neighbours ([`peers.md`](peers.md), "Member + registry and neighbours"); its own writes go out to them, others' changes arrive by + sync and relay; +* **registers** in the member registry, renews its alive stamp daily, and publishes + its stability tier; +* takes part in **remakes**: votes once it holds the content, coordinates those + proposed to it, and **switches** when one commits ([`remakes.md`](remakes.md)). A switch is a + step inside this phase, not a phase of its own: after the switch delay the node + builds the new document, verifies the epoch, replays the writes since the base into + it and swaps documents under the lock. Sync with every peer restarts; peers still + on the old epoch are told by a STATUS; +* judges its peers' liveness (dormancy), heals diverged sessions, and answers anyone; +* keeps its files up to date (above). + +## Rejoining + +Entered when enough active, ready members (two if this node was ready, one if it +never was) advertise the same newer epoch, at most 1000 generations ahead: the node +missed a switch (it was away, or its own build didn't match). With several epochs of +the newest generation advertised (two remakes committed for one epoch; [`remakes.md`](remakes.md), +Limits), it follows the one most of them are on (of those, the lowest ID). A ready node +also rejoins **sideways**, to another epoch of its own generation, when more ready +members are on it than on its own (counting itself; two at least; on a tie, the lower +ID): its own writes on the epoch it leaves all go to quarantine (step 5). What happens: + +1. at once (under the document's lock), the node swaps in an empty document labelled + with the target epoch, which takes no writes; sync with every peer resets; and a + ready node **suspends** (`Joining` again, swarm ID kept) and keeps its document as + the **stale view**: reads are answered from it, marked `x-continuum-stale`, and + writes go into it, answered as usual (and appended to the document file); a node + that wasn't ready refuses writes (`503`) as before; +2. it doesn't sync the stale view with anyone: what it takes is carried over at the + end (step 5); +3. it **downloads the epoch's snapshot** in chunks from ready members on that epoch + ([`swarm/blob.rs`](../src/swarm/blob.rs)), then ordinary sync adds the changes made since, from one source + as when joining; +4. once caught up, it **verifies** that the document starts with the target epoch's + snapshot change and that the metadata matches; +5. it decides about **its own writes** the swarm never saw: those made before the + rejoin are replayed (at their path, or where the remake's rules say) if within the + replay window of the first remake it missed, the rest go to quarantine for an + operator ([`remakes.md`](remakes.md), "Quarantine"); those made during the rejoin are replayed + whatever the window says. Either way, a write is left out where newer writes + touched its path (last writer wins, [`remakes.md`](remakes.md), "Writes around a remake"); +6. at once (under the document's lock, with step 5's replay), the node **resumes**, + drops the stale view and takes writes in the live document again; the old document + goes to the **archive** (kept as the remake's retention says). + +If no peer completes a sync within `max(dormant_after, 60 s)`, or the check in step 4 +fails, the rejoin is **abandoned**: the old document, with the writes it took +meanwhile, and its epoch are restored, the node resumes (at once, under the document's +lock), and tries again 30 s later. `/admin/epochs` shows the attempt and any error. + +```mermaid +flowchart TD + A[Active, ready members advertise a newer epoch<br/>at most 1000 generations ahead] --> A2{Enough of them for one epoch?<br/>2 if we are ready, or all our active members if fewer;<br/>else 1} + A2 -- no --> S{Ready, and more ready members on another<br/>epoch of our generation than on ours?} + S -- no --> X[Log and wait] + S -- yes --> C + A2 -- yes --> B[Target: the newest such epoch;<br/>of several in one generation, the one most of them are on] + B --> C{Retry pause over?<br/>30 s after a failure} + C -- no --> X + C -- yes --> F[At once: swap in an empty document labelled with the target epoch;<br/>if ready: suspend, the old document becomes the stale view, taking writes;<br/>reset sync with all peers, announce epoch:<br/>sync opening to ready peers on it, STATUS to the rest] + F --> F2[Download the snapshot change in chunks<br/>from ready members on the epoch; sync waits] + F2 --> G[Ordinary sync adds the changes since<br/>or the whole epoch if the download failed] + G --> H{Initial sync done?} + H -- no --> T{Longer than max of dormant_after and 60 s?} + T -- no --> G + T -- yes --> Z + H -- yes --> I{Snapshot change of the target present?} + I -- no --> G + I -- yes --> J{Snapshot metadata valid?} + J -- no --> Z[Abandon: restore old document and epoch,<br/>resume, retry later] + J -- yes --> K{Own changes missing from<br/>the epoch after ours?} + K -- none --> W + K -- "some, after a sideways rejoin" --> Q + K -- some --> L{Within that epoch's replay window?<br/>gap in epochs or time since proposal} + L -- yes --> R[Replay them, last writer wins] + L -- no, or unknown --> Q[Quarantine: /admin/quarantine] + R --> W + Q --> W[At once: replay the writes made during the rejoin,<br/>resume, drop the stale view, take writes again] + W --> O[Archive old document per retention,<br/>unless the quarantine keeps it] +``` + +## Stopping + +On SIGINT or SIGTERM (or when a transport stops: a supervised Mycelium daemon exits, +or the libp2p or Tor transport ends; `A transport stopped`, error), the node stops +accepting HTTP connections and drains the open ones (the dashboard's event streams end +with an `offline` event; from then on its keepalives say it is leaving), sends its neighbours a farewell: a last STATUS saying it isn't +ready and is leaving (one catching up from it picks another source, and each holds it +dormant at once rather than after `dormant_after`; at most 2 s), writes the state file +if the +remembered members changed, appends what the document file lacks, and exits. A second +signal exits at once. Nothing else is saved: the archive, the quarantine and all +measurements are gone. A node that stops in the middle of a rejoin loses nothing its +peers don't have, and starts the rejoin over when it returns. + +## Other phase machines + +* **A remake** goes `proposed` → `committed` or `aborted`, or `expired` when its + coordinator is gone: [`remakes.md`](remakes.md). +* **A peer**, seen from one node, has a standing (stranger, admitted, member), a + liveness (unknown, active, dormant) and a handshake state: [`peers.md`](peers.md), "Liveness, dormancy + and HELLO state". |