add documentation

1 files changed, 410 insertions, 0 deletions

diff --git a/docs/remakes.md b/docs/remakes.md
new file mode 100644
index 0000000..efcc8e3
--- /dev/null
+++ b/docs/remakes.md
@@ -0,0 +1,410 @@
+# Remakes: an operator's guide
+
+A remake moves the whole swarm to a new document, the next *epoch*. Use one to:
+
+* **compact** (`compact`): same content, no history. The document shrinks to what it
+  holds now, and joining nodes download less.
+* **replace** (`replace`): new content you supply as JSON (the shape `GET /store`
+  returns), with rules for where writes made in the meantime go.
+* **import** (`import`): new content from an Automerge document, e.g. a backup made
+  with `GET /admin/document`. Unlike `replace`, Automerge types survive (text,
+  counters, bytes, timestamps, floats).
+
+Everything else is the same for all three kinds. All endpoints need the node's basic
+auth; examples leave it out.
+
+## How a remake runs
+
+1. **Proposal.** You `POST /admin/remakes` to any node; it becomes the *coordinator*.
+   It fixes the *electorate* and, for `replace` and `import`, the *base* (the
+   document's heads now, or `baseHeads`), writes the proposal into the document and
+   votes for it.
+2. **Vote.** Every member in the electorate votes for it once the proposal reaches it,
+   after a random wait of up to half a keepalive interval, unless it already voted for
+   a competing remake of the same epoch or the remake has its majority already (then a
+   vote would only be one more change for everyone). For
+   `replace` and `import`, a member first downloads the new document from the
+   coordinator and the members that have it, so a committed remake has its document on
+   a majority.
+3. **Decision.** The coordinator commits as soon as a majority of the electorate voted
+   for the remake. It aborts if that becomes impossible, if the voting period
+   (`settle`) ends without it, or on your request. A majority that arrives after
+   the voting period doesn't count, and an abort request or a competitor's commit that
+   arrives while the coordinator builds the new document still wins. Once the remake
+   has expired (below), the coordinator doesn't commit it any more. A compaction's
+   base is fixed now: the coordinator's heads as it starts building, recorded in the
+   decision.
+4. **Switch.** After the switch delay, every node builds the new document on its own,
+   checks that it matches the committed epoch, and carries over the writes made since
+   the base that it has, its own and other nodes'. Its old document goes into its archive.
+5. **Stragglers.** Nodes that were away, or whose build didn't match, notice peers on
+   the newer epoch and *rejoin*: they download the epoch's snapshot, sync the changes
+   made since, and carry over their own writes within the replay window
+   ([`lifecycle.md`](lifecycle.md), "Rejoining").
+
+Peers on different epochs don't sync with each other, so a node stays on its epoch
+until it switches or rejoins.
+
+### Electorate and majority
+
+The electorate is the set of members whose alive stamp in the member registry is
+recent: the window is the coordinator's `--electorate-window` (default 7d, at least
+7h). Every node publishes its window in its registry entry (`window`), and renews its
+stamp every seventh of the shortest window any member publishes (once a day by
+default), at most hourly: so a coordinator with a shorter window than the others still
+counts every live member. A stamp more than a renewal interval in the future (its
+node's clock is that far ahead) doesn't count. The coordinator is always included. A remake
+needs ⌊n/2⌋+1 of the n electorate members.
+
+Each member votes for at most one remake of its epoch, and votes again only once that
+remake is aborted, so two competing remakes, even on two sides of a network partition,
+can't both get a majority. Votes are kept in the votes register
+(`_continuum/votes/<node ID>`), one entry per member.
+
+What this means in practice:
+
+* Members that have been gone for longer than the window don't count, and don't block
+  anything.
+* Members that are offline but within the window count. While more than half of the
+  electorate is reachable, they don't block anything either.
+* If half or more of the electorate is offline, no remake can commit. Wait for nodes to
+  return, or forget members that are gone for good (`DELETE /peers/{peer}`), then
+  propose again.
+* With two nodes, both must vote; with three, one can be offline.
+
+Only electorate members vote. A node that joined after the proposal doesn't.
+
+## Proposing
+
+All fields are optional:
+
+```sh
+curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' -d '{
+  "kind": "compact",
+  "settle": "8m",
+  "switchDelay": "1m",
+  "replayWindow": { "epochs": 1, "age": "7d" },
+  "archiveRetention": { "epochs": 2, "age": "30d" }
+}'
+```
+
+| Field | Default | Meaning |
+|---|---|---|
+| `kind` | `compact` | `compact`, `replace` or `import`. |
+| `spec` | none | The kind's input (below). |
+| `baseHeads` | current heads | `replace` and `import` only: what the new document accounts for; writes after it are carried over. |
+| `settle` | 8 × keepalive | Voting period, a duration ([`configuration.md`](configuration.md), "Durations and sizes"). A short one only risks an abort for lack of votes. |
+| `switchDelay` | 1 × keepalive | Wait between commit and switch, so the commit reaches every node first. Shorter only with `?force=switchDelay`. |
+| `replayWindow` | 1 epoch, 7 days | How far behind a returning node may be and still have its writes carried over automatically. |
+| `archiveRetention` | 2 epochs, 30 days | How long nodes keep the old document. |
+
+A retention is kept until *every* limit it gives has passed.
+
+`?force=` overrides checks: `true` all of them, or a comma-separated list of
+`switchDelay` (accept a shorter switch delay), `supersede` (abort expired remakes of
+this epoch and propose in their place, see below) and `clock` (propose although this
+node's clock is off from its peers').
+
+Answers:
+
+* `202` with the report and a `Location` header. With `?wait=true`, the final report
+  once decided (`200`).
+* `409` if another remake of this epoch is still open on this node's view, if an
+  expired one still holds votes (unless `?force=supersede`), or if this node can't
+  vote for a new one (it voted for another remake of this epoch, or may have before it
+  restarted), or if this node's clock is off from most of its peers' ([`measurements.md`](measurements.md),
+  Clocks): the coordinator stamps and judges the deadlines by it (unless
+  `?force=clock`).
+* `400` for an invalid request; `503` if the node isn't ready.
+
+Dropping the HTTP connection doesn't abort the remake.
+
+### Replace
+
+```json
+{
+  "kind": "replace",
+  "spec": {
+    "document": { "accounts": { "alice": { "name": "Alice" } } },
+    "rules": [
+      { "from": "users", "to": "accounts" },
+      { "from": "users/admin", "drop": true },
+      { "from": "settings", "to": "settings" }
+    ]
+  }
+}
+```
+
+The rules decide where writes made after the base go: from the old epoch, and from
+nodes that rejoin later. Paths are slash-separated strings, or arrays of segments for
+keys that contain `/`. The longest matching `from` wins, and the rest of the path is
+appended to `to`. `"drop": true` discards matching writes. A write no rule covers is
+quarantined on the node that made it. `""` is the root: `{"from": "", "to": ""}` keeps
+every path.
+
+`_continuum` (protocol data) can't be replaced or targeted by rules.
+
+To replace with something derived from the current content, read it with
+`GET /store?heads=true`, transform it, and pass the heads from `x-continuum-heads` as
+`baseHeads`: then exactly the writes you didn't see are carried over.
+
+### Import and restoring a backup
+
+Take backups with:
+
+```sh
+curl -o backup.automerge localhost:3000/admin/document
+```
+
+The file is the whole document (history included). Response headers name the swarm,
+the epoch and the heads.
+
+Restore it, keeping writes made since at their paths:
+
+```sh
+curl -X POST localhost:3000/admin/remakes \
+     -H 'Content-Type: application/octet-stream' \
+     --data-binary @backup.automerge \
+     --url-query 'rules=[{"from":"","to":""}]'
+```
+
+With `application/octet-stream`, `rules`, `baseHeads`, `settle` and
+`switchDelay` go in the query; for the retention settings use the JSON form, with
+the document base64-encoded:
+
+```sh
+base64 -w0 backup.automerge |
+  jq -Rs '{kind: "import", spec: {document: ., rules: [{from: "", to: ""}]}}' |
+  curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' --data-binary @-
+```
+
+The coordinator keeps only the content: the backup's history and its `_continuum`
+(protocol data of whatever swarm it came from) are dropped. A backup of one swarm can
+be imported into another.
+
+If no node of the swarm is left, start one from the backup instead:
+`continuum --recover-from backup.automerge …` ([`lifecycle.md`](lifecycle.md), "Starting").
+
+### A new swarm with content
+
+A new swarm can start with content instead of empty: `--genesis --remake KIND:PATH`
+makes the genesis document the way a remake of that kind would make a new epoch.
+
+```sh
+continuum --genesis --remake import:backup.automerge …  # a backup's content, of any swarm
+continuum --genesis --remake replace:seed.json …        # a JSON document, the store root
+```
+
+As with an `import` remake, only the content is kept: the backup's history and
+protocol data (its swarm ID included) are dropped, so it starts a new swarm, not the
+old one. `compact` is accepted and changes nothing.
+
+Without rules, every write made after the base is quarantined. Leaving rules out is
+right when the backup is meant to win completely.
+
+## Watching and aborting
+
+```sh
+curl localhost:3000/admin/remakes/<id>             # one report
+curl 'localhost:3000/admin/remakes/<id>?wait=true' # until decided
+curl localhost:3000/admin/remakes                  # all of them
+curl localhost:3000/admin/epochs                   # this node's epoch, archive, rejoin
+```
+
+A report has a `phase`:
+
+* `proposed`: voting.
+* `committed`: decided; nodes switch after the switch delay. It can't be undone.
+* `aborted`: the `decision.reason` says why; for missing votes it includes the tally,
+  e.g. `2 of 5 voted for it, 2 for a competing remake, 1 not yet; 3 needed`.
+* `expired`: no decision long after the voting period (`settle` plus the grace
+  of four of the coordinator's keepalives, written in the proposal as `graceMs`, so
+  every node judges it alike). The coordinator is gone, and won't commit it any more. Its
+  votes still count: the members that cast them can't vote for another remake of this
+  epoch until this one is aborted, which any node can do now (below).
+
+`votes` lists who voted, `proposal.electorate` and `proposal.quorum` who counts and
+how many are needed. `local` reports what happened on the node you asked: when it
+switched, how many paths it carried over, and the quarantine entry for writes it
+couldn't place.
+
+Abort while undecided:
+
+```sh
+curl -X DELETE localhost:3000/admin/remakes/<id>   # this one
+curl -X DELETE localhost:3000/admin/remakes        # whichever is in progress
+```
+
+The coordinator applies it within about a second (`202`, or the final report with
+`?wait=true`). If the node you ask doesn't reach the coordinator (it is in another
+partition, or gone but not yet expired), the request answers `409`: the coordinator
+could commit before the request reaches it. Ask a node that reaches it, or wait until
+the remake expires (`abortableAt`); `?force=coordinatorInactive` writes the request
+anyway. An expired remake has no coordinator to apply it, so the node you ask
+aborts it itself, under three conditions that keep it from crossing a late commit:
+
+* a whole grace period (the remake's, `graceMs`) must have passed since it expired; until
+  then only its coordinator may decide it, and the `409` says when (`abortableAt`);
+* if the node you ask sees the coordinator as an active peer, the coordinator is
+  stuck rather than gone (a different version, a bug: look at its log). Aborting then
+  needs `?force=coordinatorActive`;
+* the node you ask must have its clock in line with its peers' ([`measurements.md`](measurements.md),
+  Clocks): whether the remake expired is judged by it, and a node ahead would abort a
+  remake its coordinator may still commit. Otherwise `409`, unless `?force=clock`.
+
+A member that saw a remake expire before it voted for it says so in its log, once
+(`Didn't vote for a remake`, warn if its clock is off: it then misses remakes others
+still vote on).
+
+A committed or aborted remake answers `409`.
+
+### Superseding an expired remake
+
+Proposing while an expired remake of the epoch still holds votes answers `409` with
+what you need to decide: for each such remake its `id`, its `coordinator`, whether
+the node you asked sees that coordinator as an active peer (`coordinatorActive`),
+when it expired (`expiredAt`), from when it can be aborted (`abortableAt`) and its
+`tally`. Either abort it and propose again, or do both at once:
+
+```sh
+curl -X POST 'localhost:3000/admin/remakes?force=supersede' -H 'Content-Type: application/json' -d '{}'
+```
+
+The expired remakes are aborted ("superseded by remake …") in the same change that
+writes the new proposal, and their voters can vote for the new one. The same two
+conditions as for aborting apply; with an active coordinator, add `coordinatorActive`:
+`?force=supersede,coordinatorActive`.
+
+### Adopting a remake
+
+If a remake's coordinator is gone for good, another node can take over:
+
+```sh
+curl -X POST localhost:3000/admin/remakes/<id>/adopt
+```
+
+Only for an expired remake (or one adopted before, whose adopter is gone too). The
+node you ask becomes its coordinator (`adoptedBy` in the report). An adopted remake
+has no voting period: it commits once a majority voted for it, members that haven't
+voted yet can still do so, and it aborts only once a majority is out of reach or on
+your request.
+
+Adopt on one node only, and only when the coordinator is really gone: two nodes
+coordinating at once could decide differently.
+
+## Writes around a remake
+
+The swarm stays writable throughout.
+
+* **Writes on nodes that switch** are carried over into the new epoch, at their path
+  (compaction) or wherever the rules say.
+* **Writes on nodes that were away** are carried over when they rejoin, if the node
+  is within the replay window of the first remake it missed: at most `epochs` remakes
+  behind, or that remake was proposed at most `age` ago. Otherwise all of them are
+  quarantined, as are writes whose fate can't be determined.
+* **Last writer wins**: a write is carried over only if nothing newer touched its path,
+  a path above it or one below it, in the new epoch; newer by the times the writers'
+  clocks gave their changes. A carried-over write keeps the time it was made, so a
+  later carry-over judges by that. A write that loses is left out, logged (`Writes not
+  replayed: newer ones touched their paths`), or with `--superseded-writes quarantine`
+  kept in the quarantine. Clocks that disagree (beyond `--clock-skew-warn`) can make
+  the older write win.
+* **Granularity**: map entries are carried over per key. Lists and text are carried
+  over whole, so concurrent edits of the same list on two nodes around a remake end as
+  last-writer-wins.
+* **Who carries what**: a node that switches carries over every write made since the
+  base that it has, its own and other nodes'; a node that rejoins carries only its own
+  (the others' reached the new epoch through the nodes that switched). So a write is
+  lost only if no node that has it switches and its author can't rejoin with it: say,
+  it was made by a node cut off during the remake, synced only to nodes cut off too,
+  and its author restarted (keeping no document) before they rejoined.
+
+A rejoining node answers reads from its old document, marked with the
+`x-continuum-stale` header, and writes go into it; they are carried over into the new
+epoch when the rejoin ends, like the node's writes from before ([`lifecycle.md`](lifecycle.md),
+Rejoining).
+
+## Quarantine
+
+Writes a remake couldn't place stay on the node that made them, in memory:
+
+```sh
+curl localhost:3000/admin/quarantine                      # entries on this node
+curl localhost:3000/admin/quarantine/<id>                 # paths and values
+curl -o old.automerge localhost:3000/admin/quarantine/<id>/document
+curl -X POST localhost:3000/admin/quarantine/<id>/replay  # write them anyway
+curl -X DELETE localhost:3000/admin/quarantine/<id>       # discard
+```
+
+Replaying writes the values at their old paths as fresh writes (they win over what is
+there now). Quarantine doesn't survive a restart of that node, so look at it soon; the
+remake report's `local.quarantine` on that node names the entry.
+
+## Troubleshooting
+
+* **Aborted, "no majority in the voting period".** Too few electorate members voted in
+  time. Check which ones are missing (`proposal.electorate` minus `votes`). Retry with
+  a longer `settle` if they are only slow or briefly away; forget them if they are
+  gone for good.
+* **Aborted, "a majority is out of reach".** Members voted for a competing remake.
+  Wait until that one is decided, then propose again if still needed.
+* **Expired.** The coordinator stopped before deciding. If it restarts, it takes the
+  remake up again once ready and, the voting period being over, aborts it. If it is
+  gone for good, abort the remake, or supersede it with a new proposal
+  (`?force=supersede`).
+* **`409` on proposing.** A remake of this epoch is still open: wait for it, or abort
+  it. Or this node voted for another remake of this epoch: propose on a node that
+  didn't.
+* **A node stays on the old epoch.** It rejoins once enough ready members advertise
+  the new one ([`lifecycle.md`](lifecycle.md), "Rejoining"); `GET /admin/epochs` on it shows the
+  attempt and any error. Nodes keep trying.
+* **Several remakes committed for one epoch, or a contested remake** (`contested:
+  true`: a commit and an abort written concurrently, possible only if clocks disagree;
+  Limits, Clocks). No node switches to one on its own records (`not switching on my
+  own`, error), but it follows peers that did: the epoch most of them are on. Nodes on
+  the other epoch of that generation rejoin it sideways once they see themselves
+  outnumbered (`More peers are on another epoch of this generation than on ours`,
+  warn), and what was written there goes to quarantine ([`lifecycle.md`](lifecycle.md), Rejoining).
+  Look at those quarantines; report how it came about (Limits).
+* **Repairing records by hand.** Remake records live in the document under
+  `_continuum/remakes/<id>`. Old ones need no cleaning up: an epoch's document holds
+  the remakes proposed in it and the record of the remake that made it, and earlier
+  ones go with the previous epoch's document. Users listed in `protocol-writers` may change or delete
+  them through the store API, e.g. `curl -X DELETE …/store/_continuum/remakes/<id>`
+  to remove a broken one. Deleting a record doesn't free the votes cast for it (the
+  votes register, `_continuum/votes`, keeps them): abort it instead where you can. Every node acts on what it finds there, so such a write acts
+  on the whole swarm; it is logged. Other users get `403`, and epoch metadata
+  (`_continuum/epochs`) and the swarm ID (`_continuum/swarm`) are never writable.
+
+## Limits
+
+* **Clocks.** Voting periods, expiry and the replay window are judged by each node's
+  own clock ([`trust.md`](trust.md), What depends on clocks). With clocks further apart than the
+  grace period (four of the coordinator's keepalives), a remake aborted from another node after expiry
+  could also have been committed by its coordinator: a contested remake.
+
+* **Document size.** Large transfers go in chunks (`--blob-chunk`, default 64KiB)
+  from up to four peers at once: the new document of a `replace`/`import` remake (it
+  stays on the coordinator, and every member downloads it before voting), and the
+  epoch's snapshot for a rejoining node. One transfer is still a single message: the
+  changes made since the last remake, which reach a rejoining node in one sync
+  message. A transport drops messages it can't deliver within its delivery limit
+  (Mycelium: five minutes), which limits
+  that one to somewhere in the MiB range; remaking regularly keeps it small. Requests
+  are limited to 64 MiB.
+* **Two remakes committed for one epoch.** Majority voting rules it out while every
+  node sees the same registry and keeps its records, but three things get past it:
+  a coordinator cut off right after it committed, while the others aborted the remake
+  as expired and committed another (it switches to its own); a partition that outlasts
+  `--electorate-window`, after which each side counts only itself as the electorate;
+  and a node without `--document-file` that voted, restarted, and votes again on the
+  other side of a partition (its vote is forgotten with the records). Two epochs of
+  the next generation result; the swarm follows the one more nodes are on, and the
+  others' writes there are quarantined (Troubleshooting).
+* **Same Automerge version.** Every node must build byte-identical snapshots, so all
+  nodes need the same Automerge version. A node whose build differs rejoins instead.
+* **Lost votes.** Votes aren't stored outside the document. A node that voted, then
+  restarted before its vote reached anyone but the coordinator, doesn't see its vote.
+  So after a restart a node votes only for the one undecided remake older than its
+  start, if there is just one, and not at all while there are several. Should it still
+  vote twice, it counts for neither (the tally reports it as `for two`).