# Operations

Running a swarm from day to day: adding and removing nodes, keys, backups, upgrades,
what to alert on. How each part works is in the pages linked; this one says what to do.

## Before the first node

Settle on these, as they are hard to change later:

* **A swarm key** (`--swarm-key-file`, made with `--swarm-key-create` on the first node,
  copied to the others). Without one, any node that can reach a member may join
  ([`trust.md`](trust.md)).
* **A node key per node** (`--node-key-file`, `--node-key-create` on its first start).
  The node key is the node: a new one is a new member.
* **A state file and a document file per node** (`--state-file`, `--document-file`).
  With them a node restarts on its swarm, with its data, ready at once; without them it
  starts empty and has to join again ([`lifecycle.md`](lifecycle.md), What the files are for).
* **The transports' keys** (`--mycelium-key-file`, `--libp2p-key-file`,
  `--tor-state-dir`), so a node's addresses stay the same across restarts
  ([`transports.md`](transports.md), Keys).
* **A supervisor** that restarts the node: it stops when a transport stops (`A
  transport stopped`).

## Adding a node

Give it the swarm key, a node key to create, its files, and a seed: any member's
address (`addresses` in `GET /admin/stats` there), or nothing on a LAN with libp2p's
`--libp2p-mdns`. It joins: `503` from the store until it is ready (`/ready` says
`200` then), with `Still joining the swarm; trying on` in the log every five minutes
while it isn't, saying why. Once ready it registers itself in the member list, and the
others make it a neighbour where the topology says so ([`peers.md`](peers.md)).

## Removing a node

Stop it, then forget it from any node: `DELETE /peers/{node ID}`. That writes a
swarm-wide forget record and removes its registry entry; every node drops it. A node
that is forgotten but still alive cancels the record with its next HELLO and registers
again, so stop it first.

A node stopped but not forgotten stays a member: it is probed now and then, and it
counts in the electorate of remakes until its alive stamp is older than
`--electorate-window` (7 days by default; [`remakes.md`](remakes.md), Electorate and majority).
Several of them can keep a remake from reaching a majority: forget those that won't
come back.

## Keys

**The swarm key** can't be changed while the swarm runs: a node with another key is a
stranger to the others (`Peer no longer proves the swarm key; dropping its
membership`). Stop every node, give each the new key, and start them; with document
files they resume with their data. Nodes started one by one with the new key form a
second group that the old ones don't talk to, until all have it; they merge then, as
both sides kept writing.

**A node key** is the node's identity: a new one makes a new member. Start the node
with the new key (its document file can stay: its data is the swarm's), and forget the
old node ID.

**A transport key** (Mycelium's, libp2p's, Tor's) only moves the node's address on
that transport: peers learn the new one from its registry entry and the identity layer
(`addressChanges` in the transports' figures, [`measurements.md`](measurements.md)). Seeds given as the old
address go stale.

## Backups

`GET /admin/document` on a ready node exports the whole document with its history,
naming its swarm and epoch ([`openapi.yaml`](openapi.yaml)). Every node has all the data, so one
node's export is a backup of the swarm. A document file is a backup too, of its node,
but it is being written: export instead.

To restore:

* **into a running swarm**: an `import` remake with the backup ([`remakes.md`](remakes.md), Import
  and restoring a backup), which replaces the content for everyone;
* **a swarm that is gone**: one node with `--recover-from FILE` (ready at once, on the
  backup's swarm and epoch), the others joining it;
* **its content as a new swarm**: `--genesis --remake import:FILE`.

Quarantined writes (`GET /admin/quarantine`) and the archive of old epochs live in
memory only, and aren't in any backup.

## A whole swarm down

Nodes with document files are ready at once on restart and merge with each other. If
none has its data any more, they all join and wait for each other, and none becomes
ready: start one with `--recover-from` and a backup, or with `--genesis` (with its state
file: the same swarm again, empty), and the others join it ([`lifecycle.md`](lifecycle.md), A whole
swarm restarting).

## Upgrading

One node at a time, restarting each with its files. What to know:

* **The wire format.** Each change to it bumps a version byte, and nodes ignore
  messages of another version. Across such a change, upgraded nodes and the others
  don't talk: the swarm is split until every node is upgraded, both sides go on
  writing, and they merge then. Mycelium logs it (`Peer speaks another wire version
  (another build); ignoring its messages`, warn, once per peer); libp2p and Tor at debug
  only (`Dropping a message that doesn't decode`). Until the first release, assume
  every upgrade is such a change, and upgrade all nodes close together.
* **Automerge.** A remake needs every node to build the same snapshot, byte for byte,
  so the same Automerge version everywhere; a node with another one rejoins instead of
  switching ([`remakes.md`](remakes.md), Limits). Don't remake while the swarm runs two versions.
* **The dashboard** reloads by itself once its node serves another version of it.

## Behind a reverse proxy

The HTTP API speaks plain HTTP on `127.0.0.1:3000`; a reverse proxy in front adds TLS,
and can authenticate users itself (single sign-on, client certificates, its own
password file). It then names the user in a header, which the node believes from the
proxy's address alone (`--user-header`, `--trusted-proxies`, by default this host's
loopback addresses); requests from elsewhere still need `--htpasswd`, which becomes
optional. The proxy must set or clear that header on every request, or a client could
set it itself. With nginx:

```nginx
location / {
    auth_basic "continuum";
    auth_basic_user_file /etc/nginx/continuum.htpasswd;
    proxy_set_header X-Remote-User $remote_user;
    proxy_pass http://127.0.0.1:3000;
}
```

and the node started with `--user-header X-Remote-User`. The names the proxy gives are
the users of `--protocol-writers` and of who wrote what ([`api.md`](api.md)).

What streams (`/stream`, the dashboard's events) says `X-Accel-Buffering: no`, so nginx
passes it on as it comes. The dashboard's events have a keepalive every 15 s; `/stream`
sends nothing while nothing changes, so nginx ends a quiet one after its
`proxy_read_timeout` (60 s): raise that for it, or have clients reconnect.

## What to alert on

From the metrics (`GET /admin/stats` as Prometheus, [`measurements.md`](measurements.md)):

| Metric | Alert when | Means |
|---|---|---|
| `continuum_ready` | 0 for longer than a join takes | the node can't join or rejoin (its log says why) |
| `continuum_clock_off` | 1 | its clock is off from its peers' ([`trust.md`](trust.md), What depends on clocks) |
| `continuum_remakes_committed` | above 1 | a double commit; no node switches on its own |
| `continuum_quarantined` | above 0 | writes set aside by a rejoin, lost at the next restart |
| `continuum_catch_up_attempts` | growing | the node can't get the epoch its peers advertise |
| `continuum_reachable_members` | at most half of `continuum_members` | no remake can commit |
| `continuum_peer_heals_total` | growing steadily | sync keeps getting out of step with a peer |

And from the log, at warn or error:

* `A transport stopped`: the node stops;
* `Still joining the swarm; trying on`: with why (seeds unresolved, peers' states);
* `Several remakes committed for this epoch; not switching on my own`, `Decisions on
  this remake disagree; not switching on my own`, `Peers disagree about the newest
  epoch; following the one most of them are on`, `More peers are on another epoch of
  this generation than on ours`: two remakes committed for one epoch; look at the
  quarantines that follow ([`remakes.md`](remakes.md), Troubleshooting);
* `Quarantined writes the remake has no place for`;
* `Couldn't write the state file`, `Couldn't rewrite the document file`: a full or
  read-only disk;
* `The … file is readable by others than its owner; chmod 600 it`;
* `Peer speaks another wire version (another build); ignoring its messages`: a node on
  another build.

## What a node uses

* **Memory**: the document with its history (Automerge holds more in memory than
  `savedBytes` on disk), the archive of old epochs (`archiveRetention`, 2 epochs by
  default), quarantined writes and per-peer state. Remaking (`compact`) drops the
  history ([`remakes.md`](remakes.md)).
* **Disk**: the document file, between the full save and that plus what was appended
  since its last rewrite (`--document-rewrite`); the state file (small); Tor's state
  directory.
* **Network**: per neighbour, about two keepalives per `--keepalive` interval when
  quiet ([`measurements.md`](measurements.md), Traffic), plus sync for every write, and LINKS while a
  dashboard is open; a joining node downloads the document once (`savedBytes`, about).
* **Processes and ports**: the HTTP API (`--bind-addr`); a supervised Mycelium daemon
  and its overlay port; libp2p's listen addresses.