diff options
Diffstat (limited to 'docs/remakes.md')
| -rw-r--r-- | docs/remakes.md | 410 |
1 files changed, 410 insertions, 0 deletions
diff --git a/docs/remakes.md b/docs/remakes.md new file mode 100644 index 0000000..efcc8e3 --- /dev/null +++ b/docs/remakes.md @@ -0,0 +1,410 @@ +# Remakes: an operator's guide + +A remake moves the whole swarm to a new document, the next *epoch*. Use one to: + +* **compact** (`compact`): same content, no history. The document shrinks to what it + holds now, and joining nodes download less. +* **replace** (`replace`): new content you supply as JSON (the shape `GET /store` + returns), with rules for where writes made in the meantime go. +* **import** (`import`): new content from an Automerge document, e.g. a backup made + with `GET /admin/document`. Unlike `replace`, Automerge types survive (text, + counters, bytes, timestamps, floats). + +Everything else is the same for all three kinds. All endpoints need the node's basic +auth; examples leave it out. + +## How a remake runs + +1. **Proposal.** You `POST /admin/remakes` to any node; it becomes the *coordinator*. + It fixes the *electorate* and, for `replace` and `import`, the *base* (the + document's heads now, or `baseHeads`), writes the proposal into the document and + votes for it. +2. **Vote.** Every member in the electorate votes for it once the proposal reaches it, + after a random wait of up to half a keepalive interval, unless it already voted for + a competing remake of the same epoch or the remake has its majority already (then a + vote would only be one more change for everyone). For + `replace` and `import`, a member first downloads the new document from the + coordinator and the members that have it, so a committed remake has its document on + a majority. +3. **Decision.** The coordinator commits as soon as a majority of the electorate voted + for the remake. It aborts if that becomes impossible, if the voting period + (`settle`) ends without it, or on your request. A majority that arrives after + the voting period doesn't count, and an abort request or a competitor's commit that + arrives while the coordinator builds the new document still wins. Once the remake + has expired (below), the coordinator doesn't commit it any more. A compaction's + base is fixed now: the coordinator's heads as it starts building, recorded in the + decision. +4. **Switch.** After the switch delay, every node builds the new document on its own, + checks that it matches the committed epoch, and carries over the writes made since + the base that it has, its own and other nodes'. Its old document goes into its archive. +5. **Stragglers.** Nodes that were away, or whose build didn't match, notice peers on + the newer epoch and *rejoin*: they download the epoch's snapshot, sync the changes + made since, and carry over their own writes within the replay window + ([`lifecycle.md`](lifecycle.md), "Rejoining"). + +Peers on different epochs don't sync with each other, so a node stays on its epoch +until it switches or rejoins. + +### Electorate and majority + +The electorate is the set of members whose alive stamp in the member registry is +recent: the window is the coordinator's `--electorate-window` (default 7d, at least +7h). Every node publishes its window in its registry entry (`window`), and renews its +stamp every seventh of the shortest window any member publishes (once a day by +default), at most hourly: so a coordinator with a shorter window than the others still +counts every live member. A stamp more than a renewal interval in the future (its +node's clock is that far ahead) doesn't count. The coordinator is always included. A remake +needs ⌊n/2⌋+1 of the n electorate members. + +Each member votes for at most one remake of its epoch, and votes again only once that +remake is aborted, so two competing remakes, even on two sides of a network partition, +can't both get a majority. Votes are kept in the votes register +(`_continuum/votes/<node ID>`), one entry per member. + +What this means in practice: + +* Members that have been gone for longer than the window don't count, and don't block + anything. +* Members that are offline but within the window count. While more than half of the + electorate is reachable, they don't block anything either. +* If half or more of the electorate is offline, no remake can commit. Wait for nodes to + return, or forget members that are gone for good (`DELETE /peers/{peer}`), then + propose again. +* With two nodes, both must vote; with three, one can be offline. + +Only electorate members vote. A node that joined after the proposal doesn't. + +## Proposing + +All fields are optional: + +```sh +curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' -d '{ + "kind": "compact", + "settle": "8m", + "switchDelay": "1m", + "replayWindow": { "epochs": 1, "age": "7d" }, + "archiveRetention": { "epochs": 2, "age": "30d" } +}' +``` + +| Field | Default | Meaning | +|---|---|---| +| `kind` | `compact` | `compact`, `replace` or `import`. | +| `spec` | none | The kind's input (below). | +| `baseHeads` | current heads | `replace` and `import` only: what the new document accounts for; writes after it are carried over. | +| `settle` | 8 × keepalive | Voting period, a duration ([`configuration.md`](configuration.md), "Durations and sizes"). A short one only risks an abort for lack of votes. | +| `switchDelay` | 1 × keepalive | Wait between commit and switch, so the commit reaches every node first. Shorter only with `?force=switchDelay`. | +| `replayWindow` | 1 epoch, 7 days | How far behind a returning node may be and still have its writes carried over automatically. | +| `archiveRetention` | 2 epochs, 30 days | How long nodes keep the old document. | + +A retention is kept until *every* limit it gives has passed. + +`?force=` overrides checks: `true` all of them, or a comma-separated list of +`switchDelay` (accept a shorter switch delay), `supersede` (abort expired remakes of +this epoch and propose in their place, see below) and `clock` (propose although this +node's clock is off from its peers'). + +Answers: + +* `202` with the report and a `Location` header. With `?wait=true`, the final report + once decided (`200`). +* `409` if another remake of this epoch is still open on this node's view, if an + expired one still holds votes (unless `?force=supersede`), or if this node can't + vote for a new one (it voted for another remake of this epoch, or may have before it + restarted), or if this node's clock is off from most of its peers' ([`measurements.md`](measurements.md), + Clocks): the coordinator stamps and judges the deadlines by it (unless + `?force=clock`). +* `400` for an invalid request; `503` if the node isn't ready. + +Dropping the HTTP connection doesn't abort the remake. + +### Replace + +```json +{ + "kind": "replace", + "spec": { + "document": { "accounts": { "alice": { "name": "Alice" } } }, + "rules": [ + { "from": "users", "to": "accounts" }, + { "from": "users/admin", "drop": true }, + { "from": "settings", "to": "settings" } + ] + } +} +``` + +The rules decide where writes made after the base go: from the old epoch, and from +nodes that rejoin later. Paths are slash-separated strings, or arrays of segments for +keys that contain `/`. The longest matching `from` wins, and the rest of the path is +appended to `to`. `"drop": true` discards matching writes. A write no rule covers is +quarantined on the node that made it. `""` is the root: `{"from": "", "to": ""}` keeps +every path. + +`_continuum` (protocol data) can't be replaced or targeted by rules. + +To replace with something derived from the current content, read it with +`GET /store?heads=true`, transform it, and pass the heads from `x-continuum-heads` as +`baseHeads`: then exactly the writes you didn't see are carried over. + +### Import and restoring a backup + +Take backups with: + +```sh +curl -o backup.automerge localhost:3000/admin/document +``` + +The file is the whole document (history included). Response headers name the swarm, +the epoch and the heads. + +Restore it, keeping writes made since at their paths: + +```sh +curl -X POST localhost:3000/admin/remakes \ + -H 'Content-Type: application/octet-stream' \ + --data-binary @backup.automerge \ + --url-query 'rules=[{"from":"","to":""}]' +``` + +With `application/octet-stream`, `rules`, `baseHeads`, `settle` and +`switchDelay` go in the query; for the retention settings use the JSON form, with +the document base64-encoded: + +```sh +base64 -w0 backup.automerge | + jq -Rs '{kind: "import", spec: {document: ., rules: [{from: "", to: ""}]}}' | + curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' --data-binary @- +``` + +The coordinator keeps only the content: the backup's history and its `_continuum` +(protocol data of whatever swarm it came from) are dropped. A backup of one swarm can +be imported into another. + +If no node of the swarm is left, start one from the backup instead: +`continuum --recover-from backup.automerge …` ([`lifecycle.md`](lifecycle.md), "Starting"). + +### A new swarm with content + +A new swarm can start with content instead of empty: `--genesis --remake KIND:PATH` +makes the genesis document the way a remake of that kind would make a new epoch. + +```sh +continuum --genesis --remake import:backup.automerge … # a backup's content, of any swarm +continuum --genesis --remake replace:seed.json … # a JSON document, the store root +``` + +As with an `import` remake, only the content is kept: the backup's history and +protocol data (its swarm ID included) are dropped, so it starts a new swarm, not the +old one. `compact` is accepted and changes nothing. + +Without rules, every write made after the base is quarantined. Leaving rules out is +right when the backup is meant to win completely. + +## Watching and aborting + +```sh +curl localhost:3000/admin/remakes/<id> # one report +curl 'localhost:3000/admin/remakes/<id>?wait=true' # until decided +curl localhost:3000/admin/remakes # all of them +curl localhost:3000/admin/epochs # this node's epoch, archive, rejoin +``` + +A report has a `phase`: + +* `proposed`: voting. +* `committed`: decided; nodes switch after the switch delay. It can't be undone. +* `aborted`: the `decision.reason` says why; for missing votes it includes the tally, + e.g. `2 of 5 voted for it, 2 for a competing remake, 1 not yet; 3 needed`. +* `expired`: no decision long after the voting period (`settle` plus the grace + of four of the coordinator's keepalives, written in the proposal as `graceMs`, so + every node judges it alike). The coordinator is gone, and won't commit it any more. Its + votes still count: the members that cast them can't vote for another remake of this + epoch until this one is aborted, which any node can do now (below). + +`votes` lists who voted, `proposal.electorate` and `proposal.quorum` who counts and +how many are needed. `local` reports what happened on the node you asked: when it +switched, how many paths it carried over, and the quarantine entry for writes it +couldn't place. + +Abort while undecided: + +```sh +curl -X DELETE localhost:3000/admin/remakes/<id> # this one +curl -X DELETE localhost:3000/admin/remakes # whichever is in progress +``` + +The coordinator applies it within about a second (`202`, or the final report with +`?wait=true`). If the node you ask doesn't reach the coordinator (it is in another +partition, or gone but not yet expired), the request answers `409`: the coordinator +could commit before the request reaches it. Ask a node that reaches it, or wait until +the remake expires (`abortableAt`); `?force=coordinatorInactive` writes the request +anyway. An expired remake has no coordinator to apply it, so the node you ask +aborts it itself, under three conditions that keep it from crossing a late commit: + +* a whole grace period (the remake's, `graceMs`) must have passed since it expired; until + then only its coordinator may decide it, and the `409` says when (`abortableAt`); +* if the node you ask sees the coordinator as an active peer, the coordinator is + stuck rather than gone (a different version, a bug: look at its log). Aborting then + needs `?force=coordinatorActive`; +* the node you ask must have its clock in line with its peers' ([`measurements.md`](measurements.md), + Clocks): whether the remake expired is judged by it, and a node ahead would abort a + remake its coordinator may still commit. Otherwise `409`, unless `?force=clock`. + +A member that saw a remake expire before it voted for it says so in its log, once +(`Didn't vote for a remake`, warn if its clock is off: it then misses remakes others +still vote on). + +A committed or aborted remake answers `409`. + +### Superseding an expired remake + +Proposing while an expired remake of the epoch still holds votes answers `409` with +what you need to decide: for each such remake its `id`, its `coordinator`, whether +the node you asked sees that coordinator as an active peer (`coordinatorActive`), +when it expired (`expiredAt`), from when it can be aborted (`abortableAt`) and its +`tally`. Either abort it and propose again, or do both at once: + +```sh +curl -X POST 'localhost:3000/admin/remakes?force=supersede' -H 'Content-Type: application/json' -d '{}' +``` + +The expired remakes are aborted ("superseded by remake …") in the same change that +writes the new proposal, and their voters can vote for the new one. The same two +conditions as for aborting apply; with an active coordinator, add `coordinatorActive`: +`?force=supersede,coordinatorActive`. + +### Adopting a remake + +If a remake's coordinator is gone for good, another node can take over: + +```sh +curl -X POST localhost:3000/admin/remakes/<id>/adopt +``` + +Only for an expired remake (or one adopted before, whose adopter is gone too). The +node you ask becomes its coordinator (`adoptedBy` in the report). An adopted remake +has no voting period: it commits once a majority voted for it, members that haven't +voted yet can still do so, and it aborts only once a majority is out of reach or on +your request. + +Adopt on one node only, and only when the coordinator is really gone: two nodes +coordinating at once could decide differently. + +## Writes around a remake + +The swarm stays writable throughout. + +* **Writes on nodes that switch** are carried over into the new epoch, at their path + (compaction) or wherever the rules say. +* **Writes on nodes that were away** are carried over when they rejoin, if the node + is within the replay window of the first remake it missed: at most `epochs` remakes + behind, or that remake was proposed at most `age` ago. Otherwise all of them are + quarantined, as are writes whose fate can't be determined. +* **Last writer wins**: a write is carried over only if nothing newer touched its path, + a path above it or one below it, in the new epoch; newer by the times the writers' + clocks gave their changes. A carried-over write keeps the time it was made, so a + later carry-over judges by that. A write that loses is left out, logged (`Writes not + replayed: newer ones touched their paths`), or with `--superseded-writes quarantine` + kept in the quarantine. Clocks that disagree (beyond `--clock-skew-warn`) can make + the older write win. +* **Granularity**: map entries are carried over per key. Lists and text are carried + over whole, so concurrent edits of the same list on two nodes around a remake end as + last-writer-wins. +* **Who carries what**: a node that switches carries over every write made since the + base that it has, its own and other nodes'; a node that rejoins carries only its own + (the others' reached the new epoch through the nodes that switched). So a write is + lost only if no node that has it switches and its author can't rejoin with it: say, + it was made by a node cut off during the remake, synced only to nodes cut off too, + and its author restarted (keeping no document) before they rejoined. + +A rejoining node answers reads from its old document, marked with the +`x-continuum-stale` header, and writes go into it; they are carried over into the new +epoch when the rejoin ends, like the node's writes from before ([`lifecycle.md`](lifecycle.md), +Rejoining). + +## Quarantine + +Writes a remake couldn't place stay on the node that made them, in memory: + +```sh +curl localhost:3000/admin/quarantine # entries on this node +curl localhost:3000/admin/quarantine/<id> # paths and values +curl -o old.automerge localhost:3000/admin/quarantine/<id>/document +curl -X POST localhost:3000/admin/quarantine/<id>/replay # write them anyway +curl -X DELETE localhost:3000/admin/quarantine/<id> # discard +``` + +Replaying writes the values at their old paths as fresh writes (they win over what is +there now). Quarantine doesn't survive a restart of that node, so look at it soon; the +remake report's `local.quarantine` on that node names the entry. + +## Troubleshooting + +* **Aborted, "no majority in the voting period".** Too few electorate members voted in + time. Check which ones are missing (`proposal.electorate` minus `votes`). Retry with + a longer `settle` if they are only slow or briefly away; forget them if they are + gone for good. +* **Aborted, "a majority is out of reach".** Members voted for a competing remake. + Wait until that one is decided, then propose again if still needed. +* **Expired.** The coordinator stopped before deciding. If it restarts, it takes the + remake up again once ready and, the voting period being over, aborts it. If it is + gone for good, abort the remake, or supersede it with a new proposal + (`?force=supersede`). +* **`409` on proposing.** A remake of this epoch is still open: wait for it, or abort + it. Or this node voted for another remake of this epoch: propose on a node that + didn't. +* **A node stays on the old epoch.** It rejoins once enough ready members advertise + the new one ([`lifecycle.md`](lifecycle.md), "Rejoining"); `GET /admin/epochs` on it shows the + attempt and any error. Nodes keep trying. +* **Several remakes committed for one epoch, or a contested remake** (`contested: + true`: a commit and an abort written concurrently, possible only if clocks disagree; + Limits, Clocks). No node switches to one on its own records (`not switching on my + own`, error), but it follows peers that did: the epoch most of them are on. Nodes on + the other epoch of that generation rejoin it sideways once they see themselves + outnumbered (`More peers are on another epoch of this generation than on ours`, + warn), and what was written there goes to quarantine ([`lifecycle.md`](lifecycle.md), Rejoining). + Look at those quarantines; report how it came about (Limits). +* **Repairing records by hand.** Remake records live in the document under + `_continuum/remakes/<id>`. Old ones need no cleaning up: an epoch's document holds + the remakes proposed in it and the record of the remake that made it, and earlier + ones go with the previous epoch's document. Users listed in `protocol-writers` may change or delete + them through the store API, e.g. `curl -X DELETE …/store/_continuum/remakes/<id>` + to remove a broken one. Deleting a record doesn't free the votes cast for it (the + votes register, `_continuum/votes`, keeps them): abort it instead where you can. Every node acts on what it finds there, so such a write acts + on the whole swarm; it is logged. Other users get `403`, and epoch metadata + (`_continuum/epochs`) and the swarm ID (`_continuum/swarm`) are never writable. + +## Limits + +* **Clocks.** Voting periods, expiry and the replay window are judged by each node's + own clock ([`trust.md`](trust.md), What depends on clocks). With clocks further apart than the + grace period (four of the coordinator's keepalives), a remake aborted from another node after expiry + could also have been committed by its coordinator: a contested remake. + +* **Document size.** Large transfers go in chunks (`--blob-chunk`, default 64KiB) + from up to four peers at once: the new document of a `replace`/`import` remake (it + stays on the coordinator, and every member downloads it before voting), and the + epoch's snapshot for a rejoining node. One transfer is still a single message: the + changes made since the last remake, which reach a rejoining node in one sync + message. A transport drops messages it can't deliver within its delivery limit + (Mycelium: five minutes), which limits + that one to somewhere in the MiB range; remaking regularly keeps it small. Requests + are limited to 64 MiB. +* **Two remakes committed for one epoch.** Majority voting rules it out while every + node sees the same registry and keeps its records, but three things get past it: + a coordinator cut off right after it committed, while the others aborted the remake + as expired and committed another (it switches to its own); a partition that outlasts + `--electorate-window`, after which each side counts only itself as the electorate; + and a node without `--document-file` that voted, restarted, and votes again on the + other side of a partition (its vote is forgotten with the records). Two epochs of + the next generation result; the swarm follows the one more nodes are on, and the + others' writes there are quarantined (Troubleshooting). +* **Same Automerge version.** Every node must build byte-identical snapshots, so all + nodes need the same Automerge version. A node whose build differs rejoins instead. +* **Lost votes.** Votes aren't stored outside the document. A node that voted, then + restarted before its vote reached anyone but the coordinator, doesn't see its vote. + So after a restart a node votes only for the one undecided remake older than its + start, if there is just one, and not at all while there are several. Should it still + vote twice, it counts for neither (the tally reports it as `for two`). |