aboutsummaryrefslogtreecommitdiffstats

Operations

Running a swarm from day to day: adding and removing nodes, keys, backups, upgrades, what to alert on. How each part works is in the pages linked; this one says what to do.

Before the first node

Settle on these, as they are hard to change later:

  • A swarm key (--swarm-key-file, made with --swarm-key-create on the first node, copied to the others). Without one, any node that can reach a member may join (trust.md).
  • A node key per node (--node-key-file, --node-key-create on its first start). The node key is the node: a new one is a new member.
  • A state file and a document file per node (--state-file, --document-file). With them a node restarts on its swarm, with its data, ready at once; without them it starts empty and has to join again (lifecycle.md, What the files are for).
  • The transports' keys (--mycelium-key-file, --libp2p-key-file, --tor-state-dir), so a node's addresses stay the same across restarts (transports.md, Keys).
  • A supervisor that restarts the node: it stops when a transport stops (A transport stopped).

Adding a node

Give it the swarm key, a node key to create, its files, and a seed: any member's address (addresses in GET /admin/stats there), or nothing on a LAN with libp2p's --libp2p-mdns. It joins: 503 from the store until it is ready (/ready says 200 then), with Still joining the swarm; trying on in the log every five minutes while it isn't, saying why. Once ready it registers itself in the member list, and the others make it a neighbour where the topology says so (peers.md).

Removing a node

Stop it, then forget it from any node: DELETE /peers/{node ID}. That writes a swarm-wide forget record and removes its registry entry; every node drops it. A node that is forgotten but still alive cancels the record with its next HELLO and registers again, so stop it first.

A node stopped but not forgotten stays a member: it is probed now and then, and it counts in the electorate of remakes until its alive stamp is older than --electorate-window (7 days by default; remakes.md, Electorate and majority). Several of them can keep a remake from reaching a majority: forget those that won't come back.

Keys

The swarm key can't be changed while the swarm runs: a node with another key is a stranger to the others (Peer no longer proves the swarm key; dropping its membership). Stop every node, give each the new key, and start them; with document files they resume with their data. Nodes started one by one with the new key form a second group that the old ones don't talk to, until all have it; they merge then, as both sides kept writing.

A node key is the node's identity: a new one makes a new member. Start the node with the new key (its document file can stay: its data is the swarm's), and forget the old node ID.

A transport key (Mycelium's, libp2p's, Tor's) only moves the node's address on that transport: peers learn the new one from its registry entry and the identity layer (addressChanges in the transports' figures, measurements.md). Seeds given as the old address go stale.

Backups

GET /admin/document on a ready node exports the whole document with its history, naming its swarm and epoch (openapi.yaml). Every node has all the data, so one node's export is a backup of the swarm. A document file is a backup too, of its node, but it is being written: export instead.

To restore:

  • into a running swarm: an import remake with the backup (remakes.md, Import and restoring a backup), which replaces the content for everyone;
  • a swarm that is gone: one node with --recover-from FILE (ready at once, on the backup's swarm and epoch), the others joining it;
  • its content as a new swarm: --genesis --remake import:FILE.

Quarantined writes (GET /admin/quarantine) and the archive of old epochs live in memory only, and aren't in any backup.

A whole swarm down

Nodes with document files are ready at once on restart and merge with each other. If none has its data any more, they all join and wait for each other, and none becomes ready: start one with --recover-from and a backup, or with --genesis (with its state file: the same swarm again, empty), and the others join it (lifecycle.md, A whole swarm restarting).

Upgrading

One node at a time, restarting each with its files. What to know:

  • The wire format. Each change to it bumps a version byte, and nodes ignore messages of another version. Across such a change, upgraded nodes and the others don't talk: the swarm is split until every node is upgraded, both sides go on writing, and they merge then. Mycelium logs it (Peer speaks another wire version (another build); ignoring its messages, warn, once per peer); libp2p and Tor at debug only (Dropping a message that doesn't decode). Until the first release, assume every upgrade is such a change, and upgrade all nodes close together.
  • Automerge. A remake needs every node to build the same snapshot, byte for byte, so the same Automerge version everywhere; a node with another one rejoins instead of switching (remakes.md, Limits). Don't remake while the swarm runs two versions.
  • The dashboard reloads by itself once its node serves another version of it.

What to alert on

From the metrics (GET /admin/stats as Prometheus, measurements.md):

Metric Alert when Means
continuum_ready 0 for longer than a join takes the node can't join or rejoin (its log says why)
continuum_clock_off 1 its clock is off from its peers' (trust.md, What depends on clocks)
continuum_remakes_committed above 1 a double commit; no node switches on its own
continuum_quarantined above 0 writes set aside by a rejoin, lost at the next restart
continuum_catch_up_attempts growing the node can't get the epoch its peers advertise
continuum_reachable_members at most half of continuum_members no remake can commit
continuum_peer_heals_total growing steadily sync keeps getting out of step with a peer

And from the log, at warn or error:

  • A transport stopped: the node stops;
  • Still joining the swarm; trying on: with why (seeds unresolved, peers' states);
  • Several remakes committed for this epoch; not switching on my own, Decisions on this remake disagree; not switching on my own, Peers disagree about the newest epoch; following the one most of them are on, More peers are on another epoch of this generation than on ours: two remakes committed for one epoch; look at the quarantines that follow (remakes.md, Troubleshooting);
  • Quarantined writes the remake has no place for;
  • Couldn't write the state file, Couldn't rewrite the document file: a full or read-only disk;
  • The … file is readable by others than its owner; chmod 600 it;
  • Peer speaks another wire version (another build); ignoring its messages: a node on another build.

What a node uses

  • Memory: the document with its history (Automerge holds more in memory than savedBytes on disk), the archive of old epochs (archiveRetention, 2 epochs by default), quarantined writes and per-peer state. Remaking (compact) drops the history (remakes.md).
  • Disk: the document file, between the full save and that plus what was appended since its last rewrite (--document-rewrite); the state file (small); Tor's state directory.
  • Network: per neighbour, about two keepalives per --keepalive interval when quiet (measurements.md, Traffic), plus sync for every write, and LINKS while a dashboard is open; a joining node downloads the document once (savedBytes, about).
  • Processes and ports: the HTTP API (--bind-addr); a supervised Mycelium daemon and its overlay port; libp2p's listen addresses.