1 files changed, 149 insertions, 0 deletions
diff --git a/docs/operations.md b/docs/operations.md
new file mode 100644
index 0000000..fe09537
--- /dev/null
+++ b/docs/operations.md
@@ -0,0 +1,149 @@
+# Operations
+
+Running a swarm from day to day: adding and removing nodes, keys, backups, upgrades,
+what to alert on. How each part works is in the pages linked; this one says what to do.
+
+## Before the first node
+
+Settle on these, as they are hard to change later:
+
+* **A swarm key** (`--swarm-key-file`, made with `--swarm-key-create` on the first node,
+  copied to the others). Without one, any node that can reach a member may join
+  ([`trust.md`](trust.md)).
+* **A node key per node** (`--node-key-file`, `--node-key-create` on its first start).
+  The node key is the node: a new one is a new member.
+* **A state file and a document file per node** (`--state-file`, `--document-file`).
+  With them a node restarts on its swarm, with its data, ready at once; without them it
+  starts empty and has to join again ([`lifecycle.md`](lifecycle.md), What the files are for).
+* **The transports' keys** (`--mycelium-key-file`, `--libp2p-key-file`,
+  `--tor-state-dir`), so a node's addresses stay the same across restarts
+  ([`transports.md`](transports.md), Keys).
+* **A supervisor** that restarts the node: it stops when a transport stops (`A
+  transport stopped`).
+
+## Adding a node
+
+Give it the swarm key, a node key to create, its files, and a seed: any member's
+address (`addresses` in `GET /admin/stats` there), or nothing on a LAN with libp2p's
+`--libp2p-mdns`. It joins: `503` from the store until it is ready (`/ready` says
+`200` then), with `Still joining the swarm; trying on` in the log every five minutes
+while it isn't, saying why. Once ready it registers itself in the member list, and the
+others make it a neighbour where the topology says so ([`peers.md`](peers.md)).
+
+## Removing a node
+
+Stop it, then forget it from any node: `DELETE /peers/{node ID}`. That writes a
+swarm-wide forget record and removes its registry entry; every node drops it. A node
+that is forgotten but still alive cancels the record with its next HELLO and registers
+again, so stop it first.
+
+A node stopped but not forgotten stays a member: it is probed now and then, and it
+counts in the electorate of remakes until its alive stamp is older than
+`--electorate-window` (7 days by default; [`remakes.md`](remakes.md), Electorate and majority).
+Several of them can keep a remake from reaching a majority: forget those that won't
+come back.
+
+## Keys
+
+**The swarm key** can't be changed while the swarm runs: a node with another key is a
+stranger to the others (`Peer no longer proves the swarm key; dropping its
+membership`). Stop every node, give each the new key, and start them; with document
+files they resume with their data. Nodes started one by one with the new key form a
+second group that the old ones don't talk to, until all have it; they merge then, as
+both sides kept writing.
+
+**A node key** is the node's identity: a new one makes a new member. Start the node
+with the new key (its document file can stay: its data is the swarm's), and forget the
+old node ID.
+
+**A transport key** (Mycelium's, libp2p's, Tor's) only moves the node's address on
+that transport: peers learn the new one from its registry entry and the identity layer
+(`addressChanges` in the transports' figures, [`measurements.md`](measurements.md)). Seeds given as the old
+address go stale.
+
+## Backups
+
+`GET /admin/document` on a ready node exports the whole document with its history,
+naming its swarm and epoch ([`openapi.yaml`](openapi.yaml)). Every node has all the data, so one
+node's export is a backup of the swarm. A document file is a backup too, of its node,
+but it is being written: export instead.
+
+To restore:
+
+* **into a running swarm**: an `import` remake with the backup ([`remakes.md`](remakes.md), Import
+  and restoring a backup), which replaces the content for everyone;
+* **a swarm that is gone**: one node with `--recover-from FILE` (ready at once, on the
+  backup's swarm and epoch), the others joining it;
+* **its content as a new swarm**: `--genesis --remake import:FILE`.
+
+Quarantined writes (`GET /admin/quarantine`) and the archive of old epochs live in
+memory only, and aren't in any backup.
+
+## A whole swarm down
+
+Nodes with document files are ready at once on restart and merge with each other. If
+none has its data any more, they all join and wait for each other, and none becomes
+ready: start one with `--recover-from` and a backup, or with `--genesis` (with its state
+file: the same swarm again, empty), and the others join it ([`lifecycle.md`](lifecycle.md), A whole
+swarm restarting).
+
+## Upgrading
+
+One node at a time, restarting each with its files. What to know:
+
+* **The wire format.** Each change to it bumps a version byte, and nodes ignore
+  messages of another version. Across such a change, upgraded nodes and the others
+  don't talk: the swarm is split until every node is upgraded, both sides go on
+  writing, and they merge then. Mycelium logs it (`Peer speaks another wire version
+  (another build); ignoring its messages`, warn, once per peer); libp2p and Tor at debug
+  only (`Dropping a message that doesn't decode`). Until the first release, assume
+  every upgrade is such a change, and upgrade all nodes close together.
+* **Automerge.** A remake needs every node to build the same snapshot, byte for byte,
+  so the same Automerge version everywhere; a node with another one rejoins instead of
+  switching ([`remakes.md`](remakes.md), Limits). Don't remake while the swarm runs two versions.
+* **The dashboard** reloads by itself once its node serves another version of it.
+
+## What to alert on
+
+From the metrics (`GET /admin/stats` as Prometheus, [`measurements.md`](measurements.md)):
+
+| Metric | Alert when | Means |
+|---|---|---|
+| `continuum_ready` | 0 for longer than a join takes | the node can't join or rejoin (its log says why) |
+| `continuum_clock_off` | 1 | its clock is off from its peers' ([`trust.md`](trust.md), What depends on clocks) |
+| `continuum_remakes_committed` | above 1 | a double commit; no node switches on its own |
+| `continuum_quarantined` | above 0 | writes set aside by a rejoin, lost at the next restart |
+| `continuum_catch_up_attempts` | growing | the node can't get the epoch its peers advertise |
+| `continuum_reachable_members` | at most half of `continuum_members` | no remake can commit |
+| `continuum_peer_heals_total` | growing steadily | sync keeps getting out of step with a peer |
+
+And from the log, at warn or error:
+
+* `A transport stopped`: the node stops;
+* `Still joining the swarm; trying on`: with why (seeds unresolved, peers' states);
+* `Several remakes committed for this epoch; not switching on my own`, `Decisions on
+  this remake disagree; not switching on my own`, `Peers disagree about the newest
+  epoch; following the one most of them are on`, `More peers are on another epoch of
+  this generation than on ours`: two remakes committed for one epoch; look at the
+  quarantines that follow ([`remakes.md`](remakes.md), Troubleshooting);
+* `Quarantined writes the remake has no place for`;
+* `Couldn't write the state file`, `Couldn't rewrite the document file`: a full or
+  read-only disk;
+* `The … file is readable by others than its owner; chmod 600 it`;
+* `Peer speaks another wire version (another build); ignoring its messages`: a node on
+  another build.
+
+## What a node uses
+
+* **Memory**: the document with its history (Automerge holds more in memory than
+  `savedBytes` on disk), the archive of old epochs (`archiveRetention`, 2 epochs by
+  default), quarantined writes and per-peer state. Remaking (`compact`) drops the
+  history ([`remakes.md`](remakes.md)).
+* **Disk**: the document file, between the full save and that plus what was appended
+  since its last rewrite (`--document-rewrite`); the state file (small); Tor's state
+  directory.
+* **Network**: per neighbour, about two keepalives per `--keepalive` interval when
+  quiet ([`measurements.md`](measurements.md), Traffic), plus sync for every write, and LINKS while a
+  dashboard is open; a joining node downloads the document once (`savedBytes`, about).
+* **Processes and ports**: the HTTP API (`--bind-addr`); a supervised Mycelium daemon
+  and its overlay port; libp2p's listen addresses.