path: root/docs/remakes.md

# Remakes: an operator's guide

A remake moves the whole swarm to a new document, the next *epoch*. Use one to:

* **compact** (`compact`): same content, no history. The document shrinks to what it
  holds now, and joining nodes download less.
* **replace** (`replace`): new content you supply as JSON (the shape `GET /store`
  returns), with rules for where writes made in the meantime go.
* **import** (`import`): new content from an Automerge document, e.g. a backup made
  with `GET /admin/document`. Unlike `replace`, Automerge types survive (text,
  counters, bytes, timestamps, floats).

Everything else is the same for all three kinds. All endpoints need the node's basic
auth; examples leave it out.

## How a remake runs

1. **Proposal.** You `POST /admin/remakes` to any node; it becomes the *coordinator*.
   It fixes the *electorate* and, for `replace` and `import`, the *base* (the
   document's heads now, or `baseHeads`), writes the proposal into the document and
   votes for it.
2. **Vote.** Every member in the electorate votes for it once the proposal reaches it,
   after a random wait of up to half a keepalive interval, unless it already voted for
   a competing remake of the same epoch or the remake has its majority already (then a
   vote would only be one more change for everyone). For
   `replace` and `import`, a member first downloads the new document from the
   coordinator and the members that have it, so a committed remake has its document on
   a majority.
3. **Decision.** The coordinator commits as soon as a majority of the electorate voted
   for the remake. It aborts if that becomes impossible, if the voting period
   (`settle`) ends without it, or on your request. A majority that arrives after
   the voting period doesn't count, and an abort request or a competitor's commit that
   arrives while the coordinator builds the new document still wins. Once the remake
   has expired (below), the coordinator doesn't commit it any more. A compaction's
   base is fixed now: the coordinator's heads as it starts building, recorded in the
   decision.
4. **Switch.** After the switch delay, every node builds the new document on its own,
   checks that it matches the committed epoch, and carries over the writes made since
   the base that it has, its own and other nodes'. Its old document goes into its archive.
5. **Stragglers.** Nodes that were away, or whose build didn't match, notice peers on
   the newer epoch and *rejoin*: they download the epoch's snapshot, sync the changes
   made since, and carry over their own writes within the replay window
   ([`lifecycle.md`](lifecycle.md), "Rejoining").

Peers on different epochs don't sync with each other, so a node stays on its epoch
until it switches or rejoins.

### Electorate and majority

The electorate is the set of members whose alive stamp in the member registry is
recent: the window is the coordinator's `--electorate-window` (default 7d, at least
7h). Every node publishes its window in its registry entry (`window`), and renews its
stamp every seventh of the shortest window any member publishes (once a day by
default), at most hourly: so a coordinator with a shorter window than the others still
counts every live member. A stamp more than a renewal interval in the future (its
node's clock is that far ahead) doesn't count. The coordinator is always included. A remake
needs ⌊n/2⌋+1 of the n electorate members.

Each member votes for at most one remake of its epoch, and votes again only once that
remake is aborted, so two competing remakes, even on two sides of a network partition,
can't both get a majority. Votes are kept in the votes register
(`_continuum/votes/<node ID>`), one entry per member.

What this means in practice:

* Members that have been gone for longer than the window don't count, and don't block
  anything.
* Members that are offline but within the window count. While more than half of the
  electorate is reachable, they don't block anything either.
* If half or more of the electorate is offline, no remake can commit. Wait for nodes to
  return, or forget members that are gone for good (`DELETE /peers/{peer}`), then
  propose again.
* With two nodes, both must vote; with three, one can be offline.

Only electorate members vote. A node that joined after the proposal doesn't.

## Proposing

All fields are optional:

```sh
curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' -d '{
  "kind": "compact",
  "settle": "8m",
  "switchDelay": "1m",
  "replayWindow": { "epochs": 1, "age": "7d" },
  "archiveRetention": { "epochs": 2, "age": "30d" }
}'
```

| Field | Default | Meaning |
|---|---|---|
| `kind` | `compact` | `compact`, `replace` or `import`. |
| `spec` | none | The kind's input (below). |
| `baseHeads` | current heads | `replace` and `import` only: what the new document accounts for; writes after it are carried over. |
| `settle` | 8 × keepalive | Voting period, a duration ([`configuration.md`](configuration.md), "Durations and sizes"). A short one only risks an abort for lack of votes. |
| `switchDelay` | 1 × keepalive | Wait between commit and switch, so the commit reaches every node first. Shorter only with `?force=switchDelay`. |
| `replayWindow` | 1 epoch, 7 days | How far behind a returning node may be and still have its writes carried over automatically. |
| `archiveRetention` | 2 epochs, 30 days | How long nodes keep the old document. |

A retention is kept until *every* limit it gives has passed.

`?force=` overrides checks: `true` all of them, or a comma-separated list of
`switchDelay` (accept a shorter switch delay), `supersede` (abort expired remakes of
this epoch and propose in their place, see below) and `clock` (propose although this
node's clock is off from its peers').

Answers:

* `202` with the report and a `Location` header. With `?wait=true`, the final report
  once decided (`200`).
* `409` if another remake of this epoch is still open on this node's view, if an
  expired one still holds votes (unless `?force=supersede`), or if this node can't
  vote for a new one (it voted for another remake of this epoch, or may have before it
  restarted), or if this node's clock is off from most of its peers' ([`measurements.md`](measurements.md),
  Clocks): the coordinator stamps and judges the deadlines by it (unless
  `?force=clock`).
* `400` for an invalid request; `503` if the node isn't ready.

Dropping the HTTP connection doesn't abort the remake.

### Replace

```json
{
  "kind": "replace",
  "spec": {
    "document": { "accounts": { "alice": { "name": "Alice" } } },
    "rules": [
      { "from": "users", "to": "accounts" },
      { "from": "users/admin", "drop": true },
      { "from": "settings", "to": "settings" }
    ]
  }
}
```

The rules decide where writes made after the base go: from the old epoch, and from
nodes that rejoin later. Paths are slash-separated strings, or arrays of segments for
keys that contain `/`. The longest matching `from` wins, and the rest of the path is
appended to `to`. `"drop": true` discards matching writes. A write no rule covers is
quarantined on the node that made it. `""` is the root: `{"from": "", "to": ""}` keeps
every path.

`_continuum` (protocol data) can't be replaced or targeted by rules.

To replace with something derived from the current content, read it with
`GET /store?heads=true`, transform it, and pass the heads from `x-continuum-heads` as
`baseHeads`: then exactly the writes you didn't see are carried over.

### Import and restoring a backup

Take backups with:

```sh
curl -o backup.automerge localhost:3000/admin/document
```

The file is the whole document (history included). Response headers name the swarm,
the epoch and the heads.

Restore it, keeping writes made since at their paths:

```sh
curl -X POST localhost:3000/admin/remakes \
     -H 'Content-Type: application/octet-stream' \
     --data-binary @backup.automerge \
     --url-query 'rules=[{"from":"","to":""}]'
```

With `application/octet-stream`, `rules`, `baseHeads`, `settle` and
`switchDelay` go in the query; for the retention settings use the JSON form, with
the document base64-encoded:

```sh
base64 -w0 backup.automerge |
  jq -Rs '{kind: "import", spec: {document: ., rules: [{from: "", to: ""}]}}' |
  curl -X POST localhost:3000/admin/remakes -H 'Content-Type: application/json' --data-binary @-
```

The coordinator keeps only the content: the backup's history and its `_continuum`
(protocol data of whatever swarm it came from) are dropped. A backup of one swarm can
be imported into another.

If no node of the swarm is left, start one from the backup instead:
`continuum --recover-from backup.automerge …` ([`lifecycle.md`](lifecycle.md), "Starting").

### A new swarm with content

A new swarm can start with content instead of empty: `--genesis --remake KIND:PATH`
makes the genesis document the way a remake of that kind would make a new epoch.

```sh
continuum --genesis --remake import:backup.automerge …  # a backup's content, of any swarm
continuum --genesis --remake replace:seed.json …        # a JSON document, the store root
```

As with an `import` remake, only the content is kept: the backup's history and
protocol data (its swarm ID included) are dropped, so it starts a new swarm, not the
old one. `compact` is accepted and changes nothing.

Without rules, every write made after the base is quarantined. Leaving rules out is
right when the backup is meant to win completely.

## Watching and aborting

```sh
curl localhost:3000/admin/remakes/<id>             # one report
curl 'localhost:3000/admin/remakes/<id>?wait=true' # until decided
curl localhost:3000/admin/remakes                  # all of them
curl localhost:3000/admin/epochs                   # this node's epoch, archive, rejoin
```

A report has a `phase`:

* `proposed`: voting.
* `committed`: decided; nodes switch after the switch delay. It can't be undone.
* `aborted`: the `decision.reason` says why; for missing votes it includes the tally,
  e.g. `2 of 5 voted for it, 2 for a competing remake, 1 not yet; 3 needed`.
* `expired`: no decision long after the voting period (`settle` plus the grace
  of four of the coordinator's keepalives, written in the proposal as `graceMs`, so
  every node judges it alike). The coordinator is gone, and won't commit it any more. Its
  votes still count: the members that cast them can't vote for another remake of this
  epoch until this one is aborted, which any node can do now (below).

`votes` lists who voted, `proposal.electorate` and `proposal.quorum` who counts and
how many are needed. `local` reports what happened on the node you asked: when it
switched, how many paths it carried over, and the quarantine entry for writes it
couldn't place.

Abort while undecided:

```sh
curl -X DELETE localhost:3000/admin/remakes/<id>   # this one
curl -X DELETE localhost:3000/admin/remakes        # whichever is in progress
```

The coordinator applies it within about a second (`202`, or the final report with
`?wait=true`). If the node you ask doesn't reach the coordinator (it is in another
partition, or gone but not yet expired), the request answers `409`: the coordinator
could commit before the request reaches it. Ask a node that reaches it, or wait until
the remake expires (`abortableAt`); `?force=coordinatorInactive` writes the request
anyway. An expired remake has no coordinator to apply it, so the node you ask
aborts it itself, under three conditions that keep it from crossing a late commit:

* a whole grace period (the remake's, `graceMs`) must have passed since it expired; until
  then only its coordinator may decide it, and the `409` says when (`abortableAt`);
* if the node you ask sees the coordinator as an active peer, the coordinator is
  stuck rather than gone (a different version, a bug: look at its log). Aborting then
  needs `?force=coordinatorActive`;
* the node you ask must have its clock in line with its peers' ([`measurements.md`](measurements.md),
  Clocks): whether the remake expired is judged by it, and a node ahead would abort a
  remake its coordinator may still commit. Otherwise `409`, unless `?force=clock`.

A member that saw a remake expire before it voted for it says so in its log, once
(`Didn't vote for a remake`, warn if its clock is off: it then misses remakes others
still vote on).

A committed or aborted remake answers `409`.

### Superseding an expired remake

Proposing while an expired remake of the epoch still holds votes answers `409` with
what you need to decide: for each such remake its `id`, its `coordinator`, whether
the node you asked sees that coordinator as an active peer (`coordinatorActive`),
when it expired (`expiredAt`), from when it can be aborted (`abortableAt`) and its
`tally`. Either abort it and propose again, or do both at once:

```sh
curl -X POST 'localhost:3000/admin/remakes?force=supersede' -H 'Content-Type: application/json' -d '{}'
```

The expired remakes are aborted ("superseded by remake …") in the same change that
writes the new proposal, and their voters can vote for the new one. The same two
conditions as for aborting apply; with an active coordinator, add `coordinatorActive`:
`?force=supersede,coordinatorActive`.

### Adopting a remake

If a remake's coordinator is gone for good, another node can take over:

```sh
curl -X POST localhost:3000/admin/remakes/<id>/adopt
```

Only for an expired remake (or one adopted before, whose adopter is gone too). The
node you ask becomes its coordinator (`adoptedBy` in the report). An adopted remake
has no voting period: it commits once a majority voted for it, members that haven't
voted yet can still do so, and it aborts only once a majority is out of reach or on
your request.

Adopt on one node only, and only when the coordinator is really gone: two nodes
coordinating at once could decide differently.

## Writes around a remake

The swarm stays writable throughout.

* **Writes on nodes that switch** are carried over into the new epoch, at their path
  (compaction) or wherever the rules say.
* **Writes on nodes that were away** are carried over when they rejoin, if the node
  is within the replay window of the first remake it missed: at most `epochs` remakes
  behind, or that remake was proposed at most `age` ago. Otherwise all of them are
  quarantined, as are writes whose fate can't be determined.
* **Last writer wins**: a write is carried over only if nothing newer touched its path,
  a path above it or one below it, in the new epoch; newer by the times the writers'
  clocks gave their changes. A carried-over write keeps the time it was made, so a
  later carry-over judges by that. A write that loses is left out, logged (`Writes not
  replayed: newer ones touched their paths`), or with `--superseded-writes quarantine`
  kept in the quarantine. Clocks that disagree (beyond `--clock-skew-warn`) can make
  the older write win.
* **Granularity**: map entries are carried over per key. Lists and text are carried
  over whole, so concurrent edits of the same list on two nodes around a remake end as
  last-writer-wins.
* **Who carries what**: a node that switches carries over every write made since the
  base that it has, its own and other nodes'; a node that rejoins carries only its own
  (the others' reached the new epoch through the nodes that switched). So a write is
  lost only if no node that has it switches and its author can't rejoin with it: say,
  it was made by a node cut off during the remake, synced only to nodes cut off too,
  and its author restarted (keeping no document) before they rejoined.

A rejoining node answers reads from its old document, marked with the
`x-continuum-stale` header, and writes go into it; they are carried over into the new
epoch when the rejoin ends, like the node's writes from before ([`lifecycle.md`](lifecycle.md),
Rejoining).

## Quarantine

Writes a remake couldn't place stay on the node that made them, in memory:

```sh
curl localhost:3000/admin/quarantine                      # entries on this node
curl localhost:3000/admin/quarantine/<id>                 # paths and values
curl -o old.automerge localhost:3000/admin/quarantine/<id>/document
curl -X POST localhost:3000/admin/quarantine/<id>/replay  # write them anyway
curl -X DELETE localhost:3000/admin/quarantine/<id>       # discard
```

Replaying writes the values at their old paths as fresh writes (they win over what is
there now). Quarantine doesn't survive a restart of that node, so look at it soon; the
remake report's `local.quarantine` on that node names the entry.

## Troubleshooting

* **Aborted, "no majority in the voting period".** Too few electorate members voted in
  time. Check which ones are missing (`proposal.electorate` minus `votes`). Retry with
  a longer `settle` if they are only slow or briefly away; forget them if they are
  gone for good.
* **Aborted, "a majority is out of reach".** Members voted for a competing remake.
  Wait until that one is decided, then propose again if still needed.
* **Expired.** The coordinator stopped before deciding. If it restarts, it takes the
  remake up again once ready and, the voting period being over, aborts it. If it is
  gone for good, abort the remake, or supersede it with a new proposal
  (`?force=supersede`).
* **`409` on proposing.** A remake of this epoch is still open: wait for it, or abort
  it. Or this node voted for another remake of this epoch: propose on a node that
  didn't.
* **A node stays on the old epoch.** It rejoins once enough ready members advertise
  the new one ([`lifecycle.md`](lifecycle.md), "Rejoining"); `GET /admin/epochs` on it shows the
  attempt and any error. Nodes keep trying.
* **Several remakes committed for one epoch, or a contested remake** (`contested:
  true`: a commit and an abort written concurrently, possible only if clocks disagree;
  Limits, Clocks). No node switches to one on its own records (`not switching on my
  own`, error), but it follows peers that did: the epoch most of them are on. Nodes on
  the other epoch of that generation rejoin it sideways once they see themselves
  outnumbered (`More peers are on another epoch of this generation than on ours`,
  warn), and what was written there goes to quarantine ([`lifecycle.md`](lifecycle.md), Rejoining).
  Look at those quarantines; report how it came about (Limits).
* **Repairing records by hand.** Remake records live in the document under
  `_continuum/remakes/<id>`. Old ones need no cleaning up: an epoch's document holds
  the remakes proposed in it and the record of the remake that made it, and earlier
  ones go with the previous epoch's document. Users listed in `protocol-writers` may change or delete
  them through the store API, e.g. `curl -X DELETE …/store/_continuum/remakes/<id>`
  to remove a broken one. Deleting a record doesn't free the votes cast for it (the
  votes register, `_continuum/votes`, keeps them): abort it instead where you can. Every node acts on what it finds there, so such a write acts
  on the whole swarm; it is logged. Other users get `403`, and epoch metadata
  (`_continuum/epochs`) and the swarm ID (`_continuum/swarm`) are never writable.

## Limits

* **Clocks.** Voting periods, expiry and the replay window are judged by each node's
  own clock ([`trust.md`](trust.md), What depends on clocks). With clocks further apart than the
  grace period (four of the coordinator's keepalives), a remake aborted from another node after expiry
  could also have been committed by its coordinator: a contested remake.

* **Document size.** Large transfers go in chunks (`--blob-chunk`, default 64KiB)
  from up to four peers at once: the new document of a `replace`/`import` remake (it
  stays on the coordinator, and every member downloads it before voting), and the
  epoch's snapshot for a rejoining node. One transfer is still a single message: the
  changes made since the last remake, which reach a rejoining node in one sync
  message. A transport drops messages it can't deliver within its delivery limit
  (Mycelium: five minutes), which limits
  that one to somewhere in the MiB range; remaking regularly keeps it small. Requests
  are limited to 64 MiB.
* **Two remakes committed for one epoch.** Majority voting rules it out while every
  node sees the same registry and keeps its records, but three things get past it:
  a coordinator cut off right after it committed, while the others aborted the remake
  as expired and committed another (it switches to its own); a partition that outlasts
  `--electorate-window`, after which each side counts only itself as the electorate;
  and a node without `--document-file` that voted, restarted, and votes again on the
  other side of a partition (its vote is forgotten with the records). Two epochs of
  the next generation result; the swarm follows the one more nodes are on, and the
  others' writes there are quarantined (Troubleshooting).
* **Same Automerge version.** Every node must build byte-identical snapshots, so all
  nodes need the same Automerge version. A node whose build differs rejoins instead.
* **Lost votes.** Votes aren't stored outside the document. A node that voted, then
  restarted before its vote reached anyone but the coordinator, doesn't see its vote.
  So after a restart a node votes only for the one undecided remake older than its
  start, if there is just one, and not at all while there are several. Should it still
  vote twice, it counts for neither (the tally reports it as `for two`).