aboutsummaryrefslogtreecommitdiffstats

The life of a node

A node passes through a few phases between starting and stopping. This page says what each phase is for, how a node enters and leaves it, and what it does and doesn't do meanwhile. The protocol between peers is in peers.md, remakes in remakes.md; this page refers to them rather than repeating them.

stateDiagram-v2
    direction LR
    [*] --> Starting
    Starting --> Ready: --genesis, --recover-from,<br/>or a document file holding a document
    Starting --> Joining: otherwise
    Joining --> Ready: swarm ID known, initial sync done,<br/>no rejoin running
    Ready --> Rejoining: enough ready members<br/>advertise a newer epoch
    Rejoining --> Ready: epoch verified, own writes<br/>replayed or quarantined
    Rejoining --> Ready: abandoned - old document restored
    Ready --> Stopping: SIGINT, SIGTERM,<br/>or a transport stops
    Joining --> Stopping
    Stopping --> [*]

In the code there are two lifecycle states, Joining and Ready (NodeLifecycle); rejoining is Joining with the swarm ID kept, and starting and stopping are what main.rs does before and after. The phases below are the operator's view.

Phase Reads Writes Registers, votes, syncs as a source Files written
Starting — — no —
Joining 503 (GET /store root: the live, partial document) 503 no —
Ready live yes yes state file, document file
Rejoining the pre-rejoin document, with x-continuum-stale into the pre-rejoin document, carried over at the end no document file (the pre-rejoin document)
Stopping draining draining no state file, document file

/health, /peers, /admin/stats, /admin/config and /admin/log answer in every phase, and /ready says 200 only when ready (both without authentication); the remake endpoints and GET /admin/document need a ready node.

Starting

What happens before the node talks to anyone, in order:

  1. Configuration is loaded from defaults, files and arguments (configuration.md). Contradictions are errors; --print-config shows the result and exits.
  2. The node key (--node-key-file) and the swarm key (--swarm-key-file) are read, so a bad file fails at once. A missing key file is an error, unless --node-key-create asks for a new node key (a new node), or --swarm-key-create for a new swarm key (on the swarm's first node; the others need a copy).
  3. The state file (--state-file) is read: the swarm ID the node belongs to and members to reach it through.
  4. A document to start on, if any: the document file (--document-file) or a backup (--recover-from). It names the swarm and the epoch the node is on. A new swarm's document is made here too (--genesis --remake KIND:PATH).
  5. The transports (those enabled), in order: for Mycelium, the node connects to a running daemon (--mycelium-rest-url) or starts and supervises one; Tor bootstraps arti. A signal stops the node while they start.
  6. The workers start: message consumer, sync, peers, remakes.
  7. The HTTP API listens (--bind-addr), also while the node joins (the table above says what it answers then).

Then the node is either ready at once or joining:

Start Without a remembered swarm With one (state file)
--genesis a new swarm, ready at once that swarm again, empty; nodes of it still running merge with this one
document file holding a document ready at once, on the document's swarm and epoch the same; the two must name the same swarm
--recover-from ready at once, on the backup's swarm and epoch the same; the two must name the same swarm
seeds, or none joining through the seeds joining that swarm, through the seeds and the remembered members

Conflicts between these inputs are errors unless overridden with --force (configuration.md, "Forcing"): swarmId (the document names another swarm than the state file remembers), badDocumentFile (the file can't be read: moved aside, start without it), replaceDocumentFile (--recover-from while the document file holds a document: the backup wins). --genesis with seeds, with --recover-from, or with a document file holding a document is an error with no override.

What the files are for

  • The state file holds the swarm ID and the addresses of up to 16 members alive within the electorate window (stable ones first), and the members' aliases, so the logs name members by them from the start (perhaps stale; a change is logged once the member list says so; --log-raw-ids leaves IDs as they are). With it, a restarted node joins only its own swarm (its HELLOs claim the remembered ID) and finds it without configured seeds. Without it, a node remembers nothing, and --genesis on every start makes another new swarm while the old one's nodes go on without it. The file is written when the node becomes ready, when the remembered members change (checked every 5 minutes, written at most hourly), and at shutdown; atomically; an unreadable one is moved aside (.bad).
  • The document file holds the document itself and the actor IDs this node wrote with, so a restart resumes with the data and a later rejoin still replays the node's own writes. It is appended to every --document-append (default 5s) and rewritten after an epoch change, a restart, or when the appended part outgrew the full save (at most every --document-rewrite, default 1h); a record cut short by a crash is dropped when reading. A node starting from it is ready at once; sync merges what it missed, or it rejoins a newer epoch.
  • A backup (GET /admin/document) is the whole document with its history, naming its swarm and epoch. --recover-from starts a node on it; --genesis --remake import:FILE starts a new swarm with its content; an import remake brings its content into a running swarm (remakes.md).

A whole swarm restarting

If every node restarts without document files, none has any data: they all join, wait for each other, and none becomes ready. Start one with --genesis (with a state file: the same swarm again, empty) or with --recover-from and a backup; the others join it. Nodes with document files are ready at once and merge with each other.

Joining

The node knows no swarm (or only the remembered one) and has no document. It resolves its seeds and says HELLO; the first HELLO_REPLY tells it the swarm ID and other peers to greet. It then catches up: it syncs with one peer only, the catch-up source, so that not every peer sends it the whole document at once; the others wait and are answered once the initial sync is done (peers.md, "Handshake and join", says how the source is chosen and given up).

A joining node never gives up: seeds that don't resolve are retried with backoff, unanswered HELLOs are resent, and every five minutes it logs why it is still joining. A node without seeds and without --genesis has nothing to join and waits (seeds may be added later).

If the swarm has moved on to a newer epoch than the node's (a new node starts at genesis), the node rejoins while still joining: it downloads that epoch's snapshot first, then catches up. Bootstrap waits for the rejoin, so the node becomes ready only after the epoch check.

The node becomes ready when it knows the swarm ID, its initial sync is done (a ready peer with a non-empty document has nothing left that the node lacks) and no rejoin is running.

Ready

The normal phase. The node:

  • serves reads and writes, and syncs with its neighbours (peers.md, "Member registry and neighbours"); its own writes go out to them, others' changes arrive by sync and relay;
  • registers in the member registry, renews its alive stamp daily, and publishes its stability tier;
  • takes part in remakes: votes once it holds the content, coordinates those proposed to it, and switches when one commits (remakes.md). A switch is a step inside this phase, not a phase of its own: after the switch delay the node builds the new document, verifies the epoch, replays the writes since the base into it and swaps documents under the lock. Sync with every peer restarts; peers still on the old epoch are told by a STATUS;
  • judges its peers' liveness (dormancy), heals diverged sessions, and answers anyone;
  • keeps its files up to date (above).

Rejoining

Entered when enough active, ready members (two if this node was ready, one if it never was) advertise the same newer epoch, at most 1000 generations ahead: the node missed a switch (it was away, or its own build didn't match). With several epochs of the newest generation advertised (two remakes committed for one epoch; remakes.md, Limits), it follows the one most of them are on (of those, the lowest ID). A ready node also rejoins sideways, to another epoch of its own generation, when more ready members are on it than on its own (counting itself; two at least; on a tie, the lower ID): its own writes on the epoch it leaves all go to quarantine (step 5). What happens:

  1. at once (under the document's lock), the node swaps in an empty document labelled with the target epoch, which takes no writes; sync with every peer resets; and a ready node suspends (Joining again, swarm ID kept) and keeps its document as the stale view: reads are answered from it, marked x-continuum-stale, and writes go into it, answered as usual (and appended to the document file); a node that wasn't ready refuses writes (503) as before;
  2. it doesn't sync the stale view with anyone: what it takes is carried over at the end (step 5);
  3. it downloads the epoch's snapshot in chunks from ready members on that epoch (swarm/blob.rs), then ordinary sync adds the changes made since, from one source as when joining;
  4. once caught up, it verifies that the document starts with the target epoch's snapshot change and that the metadata matches;
  5. it decides about its own writes the swarm never saw: those made before the rejoin are replayed (at their path, or where the remake's rules say) if within the replay window of the first remake it missed, the rest go to quarantine for an operator (remakes.md, "Quarantine"); those made during the rejoin are replayed whatever the window says. Either way, a write is left out where newer writes touched its path (last writer wins, remakes.md, "Writes around a remake");
  6. at once (under the document's lock, with step 5's replay), the node resumes, drops the stale view and takes writes in the live document again; the old document goes to the archive (kept as the remake's retention says).

If no peer completes a sync within max(dormant_after, 60 s), or the check in step 4 fails, the rejoin is abandoned: the old document, with the writes it took meanwhile, and its epoch are restored, the node resumes (at once, under the document's lock), and tries again 30 s later. /admin/epochs shows the attempt and any error.

flowchart TD
    A[Active, ready members advertise a newer epoch<br/>at most 1000 generations ahead] --> A2{Enough of them for one epoch?<br/>2 if we are ready, or all our active members if fewer;<br/>else 1}
    A2 -- no --> S{Ready, and more ready members on another<br/>epoch of our generation than on ours?}
    S -- no --> X[Log and wait]
    S -- yes --> C
    A2 -- yes --> B[Target: the newest such epoch;<br/>of several in one generation, the one most of them are on]
    B --> C{Retry pause over?<br/>30 s after a failure}
    C -- no --> X
    C -- yes --> F[At once: swap in an empty document labelled with the target epoch;<br/>if ready: suspend, the old document becomes the stale view, taking writes;<br/>reset sync with all peers, announce epoch:<br/>sync opening to ready peers on it, STATUS to the rest]
    F --> F2[Download the snapshot change in chunks<br/>from ready members on the epoch; sync waits]
    F2 --> G[Ordinary sync adds the changes since<br/>or the whole epoch if the download failed]
    G --> H{Initial sync done?}
    H -- no --> T{Longer than max of dormant_after and 60 s?}
    T -- no --> G
    T -- yes --> Z
    H -- yes --> I{Snapshot change of the target present?}
    I -- no --> G
    I -- yes --> J{Snapshot metadata valid?}
    J -- no --> Z[Abandon: restore old document and epoch,<br/>resume, retry later]
    J -- yes --> K{Own changes missing from<br/>the epoch after ours?}
    K -- none --> W
    K -- "some, after a sideways rejoin" --> Q
    K -- some --> L{Within that epoch's replay window?<br/>gap in epochs or time since proposal}
    L -- yes --> R[Replay them, last writer wins]
    L -- no, or unknown --> Q[Quarantine: /admin/quarantine]
    R --> W
    Q --> W[At once: replay the writes made during the rejoin,<br/>resume, drop the stale view, take writes again]
    W --> O[Archive old document per retention,<br/>unless the quarantine keeps it]

Stopping

On SIGINT or SIGTERM (or when a transport stops: a supervised Mycelium daemon exits, or the libp2p or Tor transport ends; A transport stopped, error), the node stops accepting HTTP connections and drains the open ones (the dashboard's event streams end with an offline event; from then on its keepalives say it is leaving), sends its neighbours a farewell: a last STATUS saying it isn't ready and is leaving (one catching up from it picks another source, and each holds it dormant at once rather than after dormant_after; at most 2 s), writes the state file if the remembered members changed, appends what the document file lacks, and exits. A second signal exits at once. Nothing else is saved: the archive, the quarantine and all measurements are gone. A node that stops in the middle of a rejoin loses nothing its peers don't have, and starts the rejoin over when it returns.

Other phase machines

  • A remake goes proposed → committed or aborted, or expired when its coordinator is gone: remakes.md.
  • A peer, seen from one node, has a standing (stranger, admitted, member), a liveness (unknown, active, dormant) and a handshake state: peers.md, "Liveness, dormancy and HELLO state".