1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
|
# Operations
Running a swarm from day to day: adding and removing nodes, keys, backups, upgrades,
what to alert on. How each part works is in the pages linked; this one says what to do.
## Before the first node
Settle on these, as they are hard to change later:
* **A swarm key** (`--swarm-key-file`, made with `--swarm-key-create` on the first node,
copied to the others). Without one, any node that can reach a member may join
([`trust.md`](trust.md)).
* **A node key per node** (`--node-key-file`, `--node-key-create` on its first start).
The node key is the node: a new one is a new member.
* **A state file and a document file per node** (`--state-file`, `--document-file`).
With them a node restarts on its swarm, with its data, ready at once; without them it
starts empty and has to join again ([`lifecycle.md`](lifecycle.md), What the files are for).
* **The transports' keys** (`--mycelium-key-file`, `--libp2p-key-file`,
`--tor-state-dir`), so a node's addresses stay the same across restarts
([`transports.md`](transports.md), Keys).
* **A supervisor** that restarts the node: it stops when a transport stops (`A
transport stopped`).
## Adding a node
Give it the swarm key, a node key to create, its files, and a seed: any member's
address (`addresses` in `GET /admin/stats` there), or nothing on a LAN with libp2p's
`--libp2p-mdns`. It joins: `503` from the store until it is ready (`/ready` says
`200` then), with `Still joining the swarm; trying on` in the log every five minutes
while it isn't, saying why. Once ready it registers itself in the member list, and the
others make it a neighbour where the topology says so ([`peers.md`](peers.md)).
## Removing a node
Stop it, then forget it from any node: `DELETE /peers/{node ID}`. That writes a
swarm-wide forget record and removes its registry entry; every node drops it. A node
that is forgotten but still alive cancels the record with its next HELLO and registers
again, so stop it first.
A node stopped but not forgotten stays a member: it is probed now and then, and it
counts in the electorate of remakes until its alive stamp is older than
`--electorate-window` (7 days by default; [`remakes.md`](remakes.md), Electorate and majority).
Several of them can keep a remake from reaching a majority: forget those that won't
come back.
## Keys
**The swarm key** can't be changed while the swarm runs: a node with another key is a
stranger to the others (`Peer no longer proves the swarm key; dropping its
membership`). Stop every node, give each the new key, and start them; with document
files they resume with their data. Nodes started one by one with the new key form a
second group that the old ones don't talk to, until all have it; they merge then, as
both sides kept writing.
**A node key** is the node's identity: a new one makes a new member. Start the node
with the new key (its document file can stay: its data is the swarm's), and forget the
old node ID.
**A transport key** (Mycelium's, libp2p's, Tor's) only moves the node's address on
that transport: peers learn the new one from its registry entry and the identity layer
(`addressChanges` in the transports' figures, [`measurements.md`](measurements.md)). Seeds given as the old
address go stale.
## Backups
`GET /admin/document` on a ready node exports the whole document with its history,
naming its swarm and epoch ([`openapi.yaml`](openapi.yaml)). Every node has all the data, so one
node's export is a backup of the swarm. A document file is a backup too, of its node,
but it is being written: export instead.
To restore:
* **into a running swarm**: an `import` remake with the backup ([`remakes.md`](remakes.md), Import
and restoring a backup), which replaces the content for everyone;
* **a swarm that is gone**: one node with `--recover-from FILE` (ready at once, on the
backup's swarm and epoch), the others joining it;
* **its content as a new swarm**: `--genesis --remake import:FILE`.
Quarantined writes (`GET /admin/quarantine`) and the archive of old epochs live in
memory only, and aren't in any backup.
## A whole swarm down
Nodes with document files are ready at once on restart and merge with each other. If
none has its data any more, they all join and wait for each other, and none becomes
ready: start one with `--recover-from` and a backup, or with `--genesis` (with its state
file: the same swarm again, empty), and the others join it ([`lifecycle.md`](lifecycle.md), A whole
swarm restarting).
## Upgrading
One node at a time, restarting each with its files. What to know:
* **The wire format.** Each change to it bumps a version byte, and nodes ignore
messages of another version. Across such a change, upgraded nodes and the others
don't talk: the swarm is split until every node is upgraded, both sides go on
writing, and they merge then. Mycelium logs it (`Peer speaks another wire version
(another build); ignoring its messages`, warn, once per peer); libp2p and Tor at debug
only (`Dropping a message that doesn't decode`). Until the first release, assume
every upgrade is such a change, and upgrade all nodes close together.
* **Automerge.** A remake needs every node to build the same snapshot, byte for byte,
so the same Automerge version everywhere; a node with another one rejoins instead of
switching ([`remakes.md`](remakes.md), Limits). Don't remake while the swarm runs two versions.
* **The dashboard** reloads by itself once its node serves another version of it.
## Behind a reverse proxy
The HTTP API speaks plain HTTP on `127.0.0.1:3000`; a reverse proxy in front adds TLS,
and can authenticate users itself (single sign-on, client certificates, its own
password file). It then names the user in a header, which the node believes from the
proxy's address alone (`--user-header`, `--trusted-proxies`, by default this host's
loopback addresses); requests from elsewhere still need `--htpasswd`, which becomes
optional. The proxy must set or clear that header on every request, or a client could
set it itself. With nginx:
```nginx
location / {
auth_basic "continuum";
auth_basic_user_file /etc/nginx/continuum.htpasswd;
proxy_set_header X-Remote-User $remote_user;
proxy_pass http://127.0.0.1:3000;
}
```
and the node started with `--user-header X-Remote-User`. The names the proxy gives are
the users of `--protocol-writers` and of who wrote what ([`api.md`](api.md)).
What streams (`/stream`, the dashboard's events) says `X-Accel-Buffering: no`, so nginx
passes it on as it comes. The dashboard's events have a keepalive every 15 s; `/stream`
sends nothing while nothing changes, so nginx ends a quiet one after its
`proxy_read_timeout` (60 s): raise that for it, or have clients reconnect.
## What to alert on
From the metrics (`GET /admin/stats` as Prometheus, [`measurements.md`](measurements.md)):
| Metric | Alert when | Means |
|---|---|---|
| `continuum_ready` | 0 for longer than a join takes | the node can't join or rejoin (its log says why) |
| `continuum_clock_off` | 1 | its clock is off from its peers' ([`trust.md`](trust.md), What depends on clocks) |
| `continuum_remakes_committed` | above 1 | a double commit; no node switches on its own |
| `continuum_quarantined` | above 0 | writes set aside by a rejoin, lost at the next restart |
| `continuum_catch_up_attempts` | growing | the node can't get the epoch its peers advertise |
| `continuum_reachable_members` | at most half of `continuum_members` | no remake can commit |
| `continuum_peer_heals_total` | growing steadily | sync keeps getting out of step with a peer |
And from the log, at warn or error:
* `A transport stopped`: the node stops;
* `Still joining the swarm; trying on`: with why (seeds unresolved, peers' states);
* `Several remakes committed for this epoch; not switching on my own`, `Decisions on
this remake disagree; not switching on my own`, `Peers disagree about the newest
epoch; following the one most of them are on`, `More peers are on another epoch of
this generation than on ours`: two remakes committed for one epoch; look at the
quarantines that follow ([`remakes.md`](remakes.md), Troubleshooting);
* `Quarantined writes the remake has no place for`;
* `Couldn't write the state file`, `Couldn't rewrite the document file`: a full or
read-only disk;
* `The … file is readable by others than its owner; chmod 600 it`;
* `Peer speaks another wire version (another build); ignoring its messages`: a node on
another build.
## What a node uses
* **Memory**: the document with its history (Automerge holds more in memory than
`savedBytes` on disk), the archive of old epochs (`archiveRetention`, 2 epochs by
default), quarantined writes and per-peer state. Remaking (`compact`) drops the
history ([`remakes.md`](remakes.md)).
* **Disk**: the document file, between the full save and that plus what was appended
since its last rewrite (`--document-rewrite`); the state file (small); Tor's state
directory.
* **Network**: per neighbour, about two keepalives per `--keepalive` interval when
quiet ([`measurements.md`](measurements.md), Traffic), plus sync for every write, and LINKS while a
dashboard is open; a joining node downloads the document once (`savedBytes`, about).
* **Processes and ports**: the HTTP API (`--bind-addr`); a supervised Mycelium daemon
and its overlay port; libp2p's listen addresses.
|
