aboutsummaryrefslogtreecommitdiffstats
path: root/docs/operations.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/operations.md')
-rw-r--r--docs/operations.md49
1 files changed, 28 insertions, 21 deletions
diff --git a/docs/operations.md b/docs/operations.md
index 6aa06c2..e1b8e79 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -14,10 +14,11 @@ Settle on these, as they are hard to change later:
The node key is the node: a new one is a new member.
* **A state file and a document file per node** (`--state-file`, `--document-file`).
With them a node restarts on its swarm, with its data, ready at once; without them it
- starts empty and has to join again ([`lifecycle.md`](lifecycle.md), What the files are for).
+ starts empty and has to join again ([`lifecycle.md`](lifecycle.md),
+ [What the files are for](lifecycle.md#what-the-files-are-for)).
* **The transports' keys** (`--mycelium-key-file`, `--libp2p-key-file`,
`--tor-state-dir`), so a node's addresses stay the same across restarts
- ([`transports.md`](transports.md), Keys).
+ ([`transports.md`](transports.md), [Keys](transports.md#keys)).
* **A supervisor** that restarts the node: it stops when a transport stops (`A
transport stopped`).
@@ -39,9 +40,9 @@ again, so stop it first.
A node stopped but not forgotten stays a member: it is probed now and then, and it
counts in the electorate of remakes until its alive stamp is older than
-`--electorate-window` (7 days by default; [`remakes.md`](remakes.md), Electorate and majority).
-Several of them can keep a remake from reaching a majority: forget those that won't
-come back.
+`--electorate-window` (7 days by default; [`remakes.md`](remakes.md),
+[Electorate and majority](remakes.md#electorate-and-majority)). Several of them can keep a
+remake from reaching a majority: forget those that won't come back.
## Keys
@@ -70,8 +71,10 @@ but it is being written: export instead.
To restore:
-* **into a running swarm**: an `import` remake with the backup ([`remakes.md`](remakes.md), Import
- and restoring a backup), which replaces the content for everyone;
+* **into a running swarm**: an `import` remake with the backup
+ ([`remakes.md`](remakes.md),
+ [Import and restoring a backup](remakes.md#import-and-restoring-a-backup)), which
+ replaces the content for everyone;
* **a swarm that is gone**: one node with `--recover-from FILE` (ready at once, on the
backup's swarm and epoch), the others joining it;
* **its content as a new swarm**: `--genesis --remake import:FILE`.
@@ -84,8 +87,9 @@ memory only, and aren't in any backup.
Nodes with document files are ready at once on restart and merge with each other. If
none has its data any more, they all join and wait for each other, and none becomes
ready: start one with `--recover-from` and a backup, or with `--genesis` (with its state
-file: the same swarm again, empty), and the others join it ([`lifecycle.md`](lifecycle.md), A whole
-swarm restarting).
+file: the same swarm again, empty), and the others join it
+([`lifecycle.md`](lifecycle.md),
+[A whole swarm restarting](lifecycle.md#a-whole-swarm-restarting)).
## Upgrading
@@ -100,15 +104,16 @@ One node at a time, restarting each with its files. What to know:
every upgrade is such a change, and upgrade all nodes close together.
* **Automerge.** A remake needs every node to build the same snapshot, byte for byte,
so the same Automerge version everywhere; a node with another one rejoins instead of
- switching ([`remakes.md`](remakes.md), Limits). Don't remake while the swarm runs two versions.
+ switching ([`remakes.md`](remakes.md), [Limits](remakes.md#limits)). Don't remake while
+ the swarm runs two versions.
* **The dashboard** reloads by itself once its node serves another version of it.
## Behind a reverse proxy
The HTTP API speaks plain HTTP on `127.0.0.1:3000`; a reverse proxy in front adds TLS.
-Let the node check who users are, as it does: its `--htpasswd`, and the tokens of
-logging in on any node of the swarm ([`access.md`](access.md), Tokens). The proxy then only
-passes requests on, their `Authorization` header with them:
+Let the node check who users are, as it does: its `--htpasswd`, and the tokens of logging
+in on any node of the swarm ([`access.md`](access.md), [Tokens](access.md#tokens)). The
+proxy then only passes requests on, their `Authorization` header with them:
```sh
continuum --bind-addr unix:/run/continuum/api.sock --htpasswd htpasswd ...
@@ -219,7 +224,7 @@ From the metrics (`GET /admin/stats` as Prometheus, [`measurements.md`](measurem
| Metric | Alert when | Means |
|---|---|---|
| `continuum_ready` | 0 for longer than a join takes | the node can't join or rejoin (its log says why) |
-| `continuum_clock_off` | 1 | its clock is off from its peers' ([`trust.md`](trust.md), What depends on clocks) |
+| `continuum_clock_off` | 1 | its clock is off from its peers' ([`trust.md`](trust.md), [What depends on clocks](trust.md#what-depends-on-clocks)) |
| `continuum_remakes_committed` | above 1 | a double commit; no node switches on its own |
| `continuum_quarantined` | above 0 | writes set aside by a rejoin, lost at the next restart |
| `continuum_catch_up_attempts` | growing | the node can't get the epoch its peers advertise |
@@ -230,11 +235,12 @@ And from the log, at warn or error:
* `A transport stopped`: the node stops;
* `Still joining the swarm; trying on`: with why (seeds unresolved, peers' states);
-* `Several remakes committed for this epoch; not switching on my own`, `Decisions on
- this remake disagree; not switching on my own`, `Peers disagree about the newest
- epoch; following the one most of them are on`, `More peers are on another epoch of
- this generation than on ours`: two remakes committed for one epoch; look at the
- quarantines that follow ([`remakes.md`](remakes.md), Troubleshooting);
+* `Several remakes committed for this epoch; not switching on my own`,
+ `Decisions on this remake disagree; not switching on my own`,
+ `Peers disagree about the newest epoch; following the one most of them are on`,
+ `More peers are on another epoch of this generation than on ours`: two remakes committed
+ for one epoch; look at the quarantines that follow ([`remakes.md`](remakes.md),
+ [Troubleshooting](remakes.md#troubleshooting));
* `Quarantined writes the remake has no place for`;
* `Couldn't write the state file`, `Couldn't rewrite the document file`: a full or
read-only disk;
@@ -251,7 +257,8 @@ And from the log, at warn or error:
* **Disk**: the document file, between the full save and that plus what was appended
since its last rewrite (`--document-rewrite`); the state file (small); Tor's state
directory.
-* **Network**: per neighbour, about two keepalives per `--keepalive` interval when
- quiet ([`measurements.md`](measurements.md), Traffic), plus sync for every write; a joining node downloads the document once (`savedBytes`, about).
+* **Network**: per neighbour, about two keepalives per `--keepalive` interval when quiet
+ ([`measurements.md`](measurements.md), [Traffic](measurements.md#traffic)), plus sync
+ for every write; a joining node downloads the document once (`savedBytes`, about).
* **Processes and ports**: the HTTP API (`--bind-addr`); a supervised Mycelium daemon
and its overlay port; libp2p's listen addresses.