Remakes: an operator's guide

A remake moves the whole swarm to a new document, the next epoch. Use one to:

compact (compact): same content, no history. The document shrinks to what it holds now, and joining nodes download less.
replace (replace): new content you supply as JSON (the shape GET /store returns), with rules for where writes made in the meantime go.
import (import): new content from an Automerge document, e.g. a backup made with GET /admin/document. Unlike replace, Automerge types survive (text, counters, bytes, timestamps, floats).

Everything else is the same for all three kinds. All endpoints need the node's basic auth; examples leave it out.

How a remake runs

Proposal. You POST /admin/remakes to any node; it becomes the coordinator. It fixes the electorate and, for replace and import, the base (the document's heads now, or baseHeads), writes the proposal into the document and votes for it.
Vote. Every member in the electorate votes for it once the proposal reaches it, after a random wait of up to half a keepalive interval, unless it already voted for a competing remake of the same epoch or the remake has its majority already (then a vote would only be one more change for everyone). For replace and import, a member first downloads the new document from the coordinator and the members that have it, so a committed remake has its document on a majority.
Decision. The coordinator commits as soon as a majority of the electorate voted for the remake. It aborts if that becomes impossible, if the voting period (settle) ends without it, or on your request. A majority that arrives after the voting period doesn't count, and an abort request or a competitor's commit that arrives while the coordinator builds the new document still wins. Once the remake has expired (below), the coordinator doesn't commit it any more. A compaction's base is fixed now: the coordinator's heads as it starts building, recorded in the decision.
Switch. After the switch delay, every node builds the new document on its own, checks that it matches the committed epoch, and carries over the writes made since the base that it has, its own and other nodes'. Its old document goes into its archive.
Stragglers. Nodes that were away, or whose build didn't match, notice peers on the newer epoch and rejoin: they download the epoch's snapshot, sync the changes made since, and carry over their own writes within the replay window (lifecycle.md, "Rejoining").

Peers on different epochs don't sync with each other, so a node stays on its epoch until it switches or rejoins.

Electorate and majority

The electorate is the set of members whose alive stamp in the member registry is recent: the window is the coordinator's --electorate-window (default 7d, at least 7h). Every node publishes its window in its registry entry (window), and renews its stamp every seventh of the shortest window any member publishes (once a day by default), at most hourly: so a coordinator with a shorter window than the others still counts every live member. A stamp more than a renewal interval in the future (its node's clock is that far ahead) doesn't count. The coordinator is always included. A remake needs ⌊n/2⌋+1 of the n electorate members.

Each member votes for at most one remake of its epoch, and votes again only once that remake is aborted, so two competing remakes, even on two sides of a network partition, can't both get a majority. Votes are kept in the votes register (_continuum/votes/<node ID>), one entry per member.

What this means in practice:

Members that have been gone for longer than the window don't count, and don't block anything.
Members that are offline but within the window count. While more than half of the electorate is reachable, they don't block anything either.
If half or more of the electorate is offline, no remake can commit. Wait for nodes to return, or forget members that are gone for good (DELETE /peers/{peer}), then propose again.
With two nodes, both must vote; with three, one can be offline.

Only electorate members vote. A node that joined after the proposal doesn't.

Proposing

All fields are optional:

curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' -d '{
  "kind": "compact",
  "settle": "8m",
  "switchDelay": "1m",
  "replayWindow": { "epochs": 1, "age": "7d" },
  "archiveRetention": { "epochs": 2, "age": "30d" }
}'

Field	Default	Meaning
`kind`	`compact`	`compact`, `replace` or `import`.
`spec`	none	The kind's input (below).
`baseHeads`	current heads	`replace` and `import` only: what the new document accounts for; writes after it are carried over.
`settle`	8 × keepalive	Voting period, a duration (`configuration.md`, "Durations and sizes"). A short one only risks an abort for lack of votes.
`switchDelay`	1 × keepalive	Wait between commit and switch, so the commit reaches every node first. Shorter only with `?force=switchDelay`.
`replayWindow`	1 epoch, 7 days	How far behind a returning node may be and still have its writes carried over automatically.
`archiveRetention`	2 epochs, 30 days	How long nodes keep the old document.

A retention is kept until every limit it gives has passed.

?force= overrides checks: true all of them, or a comma-separated list of switchDelay (accept a shorter switch delay), supersede (abort expired remakes of this epoch and propose in their place, see below) and clock (propose although this node's clock is off from its peers').

Answers:

202 with the report and a Location header. With ?wait=true, the final report once decided (200).
409 if another remake of this epoch is still open on this node's view, if an expired one still holds votes (unless ?force=supersede), or if this node can't vote for a new one (it voted for another remake of this epoch, or may have before it restarted), or if this node's clock is off from most of its peers' (measurements.md, Clocks): the coordinator stamps and judges the deadlines by it (unless ?force=clock).
400 for an invalid request; 503 if the node isn't ready.

Dropping the HTTP connection doesn't abort the remake.

Replace

{
  "kind": "replace",
  "spec": {
    "document": { "accounts": { "alice": { "name": "Alice" } } },
    "rules": [
      { "from": "users", "to": "accounts" },
      { "from": "users/admin", "drop": true },
      { "from": "settings", "to": "settings" }
    ]
  }
}

The rules decide where writes made after the base go: from the old epoch, and from nodes that rejoin later. Paths are slash-separated strings, or arrays of segments for keys that contain /. The longest matching from wins, and the rest of the path is appended to to. "drop": true discards matching writes. A write no rule covers is quarantined on the node that made it. "" is the root: {"from": "", "to": ""} keeps every path.

_continuum (protocol data) can't be replaced or targeted by rules.

To replace with something derived from the current content, read it with GET /store?heads=true, transform it, and pass the heads from x-continuum-heads as baseHeads: then exactly the writes you didn't see are carried over.

Import and restoring a backup

Take backups with:

curl -o backup.automerge localhost:3000/admin/document

The file is the whole document (history included). Response headers name the swarm, the epoch and the heads.

Restore it, keeping writes made since at their paths:

curl -X POST localhost:3000/admin/remakes \
     -H 'Content-Type: application/octet-stream' \
     --data-binary @backup.automerge \
     --url-query 'rules=[{"from":"","to":""}]'

With application/octet-stream, rules, baseHeads, settle and switchDelay go in the query; for the retention settings use the JSON form, with the document base64-encoded:

base64 -w0 backup.automerge |
  jq -Rs '{kind: "import", spec: {document: ., rules: [{from: "", to: ""}]}}' |
  curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' --data-binary @-

The coordinator keeps only the content: the backup's history and its _continuum (protocol data of whatever swarm it came from) are dropped. A backup of one swarm can be imported into another.

If no node of the swarm is left, start one from the backup instead: continuum --recover-from backup.automerge … (lifecycle.md, "Starting").

A new swarm with content

A new swarm can start with content instead of empty: --genesis --remake KIND:PATH makes the genesis document the way a remake of that kind would make a new epoch.

continuum --genesis --remake import:backup.automerge …  # a backup's content, of any swarm
continuum --genesis --remake replace:seed.json …        # a JSON document, the store root

As with an import remake, only the content is kept: the backup's history and protocol data (its swarm ID included) are dropped, so it starts a new swarm, not the old one. compact is accepted and changes nothing.

Without rules, every write made after the base is quarantined. Leaving rules out is right when the backup is meant to win completely.

Watching and aborting

curl localhost:3000/admin/remakes/<id>             # one report
curl 'localhost:3000/admin/remakes/<id>?wait=true' # until decided
curl localhost:3000/admin/remakes                  # all of them
curl localhost:3000/admin/epochs                   # this node's epoch, archive, rejoin

A report has a phase:

proposed: voting.
committed: decided; nodes switch after the switch delay. It can't be undone.
aborted: the decision.reason says why; for missing votes it includes the tally, e.g. 2 of 5 voted for it, 2 for a competing remake, 1 not yet; 3 needed.
expired: no decision long after the voting period (settle plus the grace of four of the coordinator's keepalives, written in the proposal as graceMs, so every node judges it alike). The coordinator is gone, and won't commit it any more. Its votes still count: the members that cast them can't vote for another remake of this epoch until this one is aborted, which any node can do now (below).

votes lists who voted, proposal.electorate and proposal.quorum who counts and how many are needed. local reports what happened on the node you asked: when it switched, how many paths it carried over, and the quarantine entry for writes it couldn't place.

Abort while undecided:

curl -X DELETE localhost:3000/admin/remakes/<id>   # this one
curl -X DELETE localhost:3000/admin/remakes        # whichever is in progress

The coordinator applies it within about a second (202, or the final report with ?wait=true). If the node you ask doesn't reach the coordinator (it is in another partition, or gone but not yet expired), the request answers 409: the coordinator could commit before the request reaches it. Ask a node that reaches it, or wait until the remake expires (abortableAt); ?force=coordinatorInactive writes the request anyway. An expired remake has no coordinator to apply it, so the node you ask aborts it itself, under three conditions that keep it from crossing a late commit:

a whole grace period (the remake's, graceMs) must have passed since it expired; until then only its coordinator may decide it, and the 409 says when (abortableAt);
if the node you ask sees the coordinator as an active peer, the coordinator is stuck rather than gone (a different version, a bug: look at its log). Aborting then needs ?force=coordinatorActive;
the node you ask must have its clock in line with its peers' (measurements.md, Clocks): whether the remake expired is judged by it, and a node ahead would abort a remake its coordinator may still commit. Otherwise 409, unless ?force=clock.

A member that saw a remake expire before it voted for it says so in its log, once (Didn't vote for a remake, warn if its clock is off: it then misses remakes others still vote on).

A committed or aborted remake answers 409.

Superseding an expired remake

Proposing while an expired remake of the epoch still holds votes answers 409 with what you need to decide: for each such remake its id, its coordinator, whether the node you asked sees that coordinator as an active peer (coordinatorActive), when it expired (expiredAt), from when it can be aborted (abortableAt) and its tally. Either abort it and propose again, or do both at once:

curl -X POST 'localhost:3000/admin/remakes?force=supersede' -H 'Content-Type: application/json' -d '{}'

The expired remakes are aborted ("superseded by remake …") in the same change that writes the new proposal, and their voters can vote for the new one. The same two conditions as for aborting apply; with an active coordinator, add coordinatorActive: ?force=supersede,coordinatorActive.

Adopting a remake

If a remake's coordinator is gone for good, another node can take over:

curl -X POST localhost:3000/admin/remakes/<id>/adopt

Only for an expired remake (or one adopted before, whose adopter is gone too). The node you ask becomes its coordinator (adoptedBy in the report). An adopted remake has no voting period: it commits once a majority voted for it, members that haven't voted yet can still do so, and it aborts only once a majority is out of reach or on your request.

Adopt on one node only, and only when the coordinator is really gone: two nodes coordinating at once could decide differently.

Writes around a remake

The swarm stays writable throughout.

Writes on nodes that switch are carried over into the new epoch, at their path (compaction) or wherever the rules say.
Writes on nodes that were away are carried over when they rejoin, if the node is within the replay window of the first remake it missed: at most epochs remakes behind, or that remake was proposed at most age ago. Otherwise all of them are quarantined, as are writes whose fate can't be determined.
Last writer wins: a write is carried over only if nothing newer touched its path, a path above it or one below it, in the new epoch; newer by the times the writers' clocks gave their changes. A carried-over write keeps the time it was made, so a later carry-over judges by that. A write that loses is left out, logged (Writes not replayed: newer ones touched their paths), or with --superseded-writes quarantine kept in the quarantine. Clocks that disagree (beyond --clock-skew-warn) can make the older write win.
Granularity: map entries are carried over per key. Lists and text are carried over whole, so concurrent edits of the same list on two nodes around a remake end as last-writer-wins.
Who carries what: a node that switches carries over every write made since the base that it has, its own and other nodes'; a node that rejoins carries only its own (the others' reached the new epoch through the nodes that switched). So a write is lost only if no node that has it switches and its author can't rejoin with it: say, it was made by a node cut off during the remake, synced only to nodes cut off too, and its author restarted (keeping no document) before they rejoined.

A rejoining node answers reads from its old document, marked with the x-continuum-stale header, and writes go into it; they are carried over into the new epoch when the rejoin ends, like the node's writes from before (lifecycle.md, Rejoining).

Quarantine

Writes a remake couldn't place stay on the node that made them, in memory:

curl localhost:3000/admin/quarantine                      # entries on this node
curl localhost:3000/admin/quarantine/<id>                 # paths and values
curl -o old.automerge localhost:3000/admin/quarantine/<id>/document
curl -X POST localhost:3000/admin/quarantine/<id>/replay  # write them anyway
curl -X DELETE localhost:3000/admin/quarantine/<id>       # discard

Replaying writes the values at their old paths as fresh writes (they win over what is there now). Quarantine doesn't survive a restart of that node, so look at it soon; the remake report's local.quarantine on that node names the entry.

Troubleshooting

Aborted, "no majority in the voting period". Too few electorate members voted in time. Check which ones are missing (proposal.electorate minus votes). Retry with a longer settle if they are only slow or briefly away; forget them if they are gone for good.
Aborted, "a majority is out of reach". Members voted for a competing remake. Wait until that one is decided, then propose again if still needed.
Expired. The coordinator stopped before deciding. If it restarts, it takes the remake up again once ready and, the voting period being over, aborts it. If it is gone for good, abort the remake, or supersede it with a new proposal (?force=supersede).
409 on proposing. A remake of this epoch is still open: wait for it, or abort it. Or this node voted for another remake of this epoch: propose on a node that didn't.
A node stays on the old epoch. It rejoins once enough ready members advertise the new one (lifecycle.md, "Rejoining"); GET /admin/epochs on it shows the attempt and any error. Nodes keep trying.
Several remakes committed for one epoch, or a contested remake (contested: true: a commit and an abort written concurrently, possible only if clocks disagree; Limits, Clocks). No node switches to one on its own records (not switching on my own, error), but it follows peers that did: the epoch most of them are on. Nodes on the other epoch of that generation rejoin it sideways once they see themselves outnumbered (More peers are on another epoch of this generation than on ours, warn), and what was written there goes to quarantine (lifecycle.md, Rejoining). Look at those quarantines; report how it came about (Limits).
Repairing records by hand. Remake records live in the document under _continuum/remakes/<id>. Old ones need no cleaning up: an epoch's document holds the remakes proposed in it and the record of the remake that made it, and earlier ones go with the previous epoch's document. Users listed in protocol-writers may change or delete them through the store API, e.g. curl -X DELETE …/store/_continuum/remakes/<id> to remove a broken one. Deleting a record doesn't free the votes cast for it (the votes register, _continuum/votes, keeps them): abort it instead where you can. Every node acts on what it finds there, so such a write acts on the whole swarm; it is logged. Other users get 403, and epoch metadata (_continuum/epochs) and the swarm ID (_continuum/swarm) are never writable.

Limits

Clocks. Voting periods, expiry and the replay window are judged by each node's own clock (trust.md, What depends on clocks). With clocks further apart than the grace period (four of the coordinator's keepalives), a remake aborted from another node after expiry could also have been committed by its coordinator: a contested remake.
Document size. Large transfers go in chunks (--blob-chunk, default 64KiB) from up to four peers at once: the new document of a replace/import remake (it stays on the coordinator, and every member downloads it before voting), and the epoch's snapshot for a rejoining node. One transfer is still a single message: the changes made since the last remake, which reach a rejoining node in one sync message. A transport drops messages it can't deliver within its delivery limit (Mycelium: five minutes), which limits that one to somewhere in the MiB range; remaking regularly keeps it small. Requests are limited to 64 MiB.
Two remakes committed for one epoch. Majority voting rules it out while every node sees the same registry and keeps its records, but three things get past it: a coordinator cut off right after it committed, while the others aborted the remake as expired and committed another (it switches to its own); a partition that outlasts --electorate-window, after which each side counts only itself as the electorate; and a node without --document-file that voted, restarted, and votes again on the other side of a partition (its vote is forgotten with the records). Two epochs of the next generation result; the swarm follows the one more nodes are on, and the others' writes there are quarantined (Troubleshooting).
Same Automerge version. Every node must build byte-identical snapshots, so all nodes need the same Automerge version. A node whose build differs rejoins instead.
Lost votes. Votes aren't stored outside the document. A node that voted, then restarted before its vote reached anyone but the coordinator, doesn't see its vote. So after a restart a node votes only for the one undecided remake older than its start, if there is just one, and not at all while there are several. Should it still vote twice, it counts for neither (the tally reports it as for two).