diff options
| author | tv <tv@krebsco.de> | 2026-11-18 18:49:50 +0000 |
|---|---|---|
| committer | tv <tv@krebsco.de> | 2026-11-18 20:29:41 +0100 |
| commit | c64fcc05d8c53bee99ab039bd709039afb730cdd (patch) | |
| tree | 45baad78d7b92de42283be2954fc3f9b4dd6129d /docs | |
| parent | 598366b03679eac9d04f32d9194f3ba4ba9c3da8 (diff) | |
federation: a node stops at a version it doesn't read
A version event writing a proposal passed on the version a node reads, which doesn't
read as a version there (of a later federation, using what a later continuum takes, or
none at all), the node can't judge: `Fold::stops`. The store stops at it: it takes
nothing and refuses nothing from then on, holding back whatever comes, that event first
(`Offered::Stopped`), and keeps none of it across a restart, where it stops there again.
One beside the version the node reads is refused of its own, as any other.
Its operator has it drop that event (`POST /admin/events/drop`), or one it builds on:
it is refused as dropped, or as built on what was, any version event writing that
version with it, kept in the store file; the node judges what it held back and goes on
under the version before. Allowed again, it stops there once more.
The node logs the stop and its end; `GET /events` names the event, why, and how many it
holds back (`stopped`); a put answers `stopped`, written into the document all the same;
a node voting by itself casts nothing while stopped. The test fold's eras tagged `new`
stop a store.
Row 39 is tested (`federation_attack_39`); of rows 9, 10 and 48, a node stops at the
version event.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/federation.md | 44 | ||||
| -rw-r--r-- | docs/guides/federation/proposing-and-voting.md | 5 | ||||
| -rw-r--r-- | docs/guides/federation/refusals-and-drops.md | 8 | ||||
| -rw-r--r-- | docs/guides/federation/upgrades.md | 11 | ||||
| -rw-r--r-- | docs/guides/federation/watching.md | 8 | ||||
| -rw-r--r-- | docs/guides/federation/when-things-go-wrong.md | 34 | ||||
| -rw-r--r-- | docs/guides/federation/writing-and-reading.md | 5 | ||||
| -rw-r--r-- | docs/openapi.yaml | 40 | ||||
| -rw-r--r-- | docs/store.md | 32 |
9 files changed, 142 insertions, 45 deletions
diff --git a/docs/federation.md b/docs/federation.md index 190f2b4..cef373b 100644 --- a/docs/federation.md +++ b/docs/federation.md @@ -242,12 +242,15 @@ application keeps party names apart from its spelling (no party named `wiki` whi A version is none if a field is missing or more (but for `checkpoint`, which it may leave out), if it gives a party more keys than a key set may hold or more nodes than a node list, or a rule a path longer than a write's may be, or if it breaks any of the -above. It can't pass, and a version event naming it is refused. A proposal is taken -whatever it holds: whether it is a version is asked where a version event names it, so -that nodes knowing more of what a version may say don't part from those knowing less. A -node shows whether a proposal is a version by the federation it knows, and a party's -client backs none that isn't: two thirds backing one that can never be written bind the -federation to its version, as a yes binds ([Changing the policy](#changing-the-policy)). +above. A proposal is taken whatever it holds: whether it is a version is asked where a +version event names it, so that nodes knowing more of what a version may say don't part +from those knowing less. A node can't tell a version that is none from one a later +continuum reads: at a version event writing one that passed, it stops +([Changing the federation](#changing-the-federation)). A node shows whether a proposal is +a version by the federation it knows, and a party's client backs none that isn't: two +thirds backing one that is none stop every node at its version event, until their +operators have them drop it, and bind the federation to its version, as a yes binds +([Changing the policy](#changing-the-policy)). What the swarm's document holds beyond the events, and what a node takes at its API, is the store's ([`store.md`](store.md), [On Automerge](store.md#on-automerge)). @@ -296,8 +299,10 @@ Once a proposal passed on the current version, a node writes it as soon as it ho passed, by a **version** event, `{"deps": […], "version": <hash>}`, which no party signs: any node may write it, and the same version written by several is one (below). Every node judges it at its history: the proposal proposed there on the version current -there, passed there, and a version changing that one. A version takes effect where it is -written: it is current at every history that holds it and no version after it. +there, passed there, and a version changing that one; where the proposal is none to the +node, it can't judge it ([Changing the federation](#changing-the-federation)). A version +takes effect where it is written: it is current at every history that holds it and no +version after it. **A node takes a version only where it changes the version the node reads**, or is one it took, written again. A version event of any other, valid at its history or not, it @@ -519,7 +524,9 @@ rule for checkpoints, none for a version that changes more than its number, what changes and its checkpoint, which it checks against the current one, nor for one whose heads leave out what it held for a while ([Checkpoints](#checkpoints)). Where two checkpoints are proposed at once, the votes split, as any do; a rule may vote `false` on -the one it doesn't back, so that one dies and its backers are free. +the one it doesn't back, so that one dies and its backers are free. A node that stopped +at a version it doesn't read casts nothing, by itself or by hand, as what it reads is +behind ([Changing the federation](#changing-the-federation)). It signs with a key the party's version names, which the party keeps on that node, not apart: one stolen there votes, and writes the party's key set and node list, as the @@ -659,7 +666,7 @@ fold, with crafted events. *Sim*: a simulation. *Bench*: measured, not judged. | 6 | A key set signed by a key the version doesn't name, or by one a later version replaced, at a history from before | Refused; or, written under a version before the one that named the party's keys anew, it counts no more | unit | | 7 | A key set without the key that signed it | Counts: a set is checked where it is written | unit | | 8 | A version event without the votes, or for a proposal not on the version current at its history | Refused at its history | unit | -| 9 | A version with a field missing or more, a rule under `_continuum`, without a literal first segment, with `**` not last, `{party}` twice, or a writer that is no party of it | No version: can't pass; a version event naming it is refused | unit | +| 9 | A version with a field missing or more, a rule under `_continuum`, without a literal first segment, with `**` not last, `{party}` twice, or a writer that is no party of it | No version: passed, a node stops at a version event writing it, as at one it doesn't know (39) | unit | | 10 | A proposal naming a federation, or saying what, a node doesn't know | Taken as a proposal; asked where a version event names it | unit | | 11 | `true` on two proposals of one version | The second refused, unless the first is dead at its history | unit | | 12 | `true` on two proposals at once, from two nodes | Both taken; two versions changing one may be written; each node keeps the one it took and refuses the other; reported; the parties fork | unit, every order | @@ -689,7 +696,7 @@ fold, with crafted events. *Sim*: a simulation. *Bench*: measured, not judged. | 36 | A fork's version giving a party that didn't go keys of its founder's | Joiners holding the federation are told, party by party | unit | | 37 | Two founders forking one version at one set of heads, or starting a federation from one first version | One store: its root is built byte for byte | unit | | 38 | A group whose parties list no nodes forks | Its version names their node lists | unit | -| 39 | A federation, or a version, a node doesn't know | It stops taking events | unit | +| 39 | A federation, or a version, a node doesn't know | Of its root, it holds no store; at a version event writing one that passed, it stops: takes nothing and refuses nothing, until it runs a continuum that reads it, or its operator has it drop that event | unit | | 40 | Two parties each writing a key set beside the other's vote, crossed | Nothing: votes go by their version's keys | unit | | 41 | A federation's root with heads or events kept, or whose version lists no node | None; or nothing after it is taken, as no node is a member | unit | | 42 | A version replacing every listed party's keys | The lists still count, each its party's until it writes another | unit | @@ -698,7 +705,7 @@ fold, with crafted events. *Sim*: a simulation. *Bench*: measured, not judged. | 45 | An honest node writes a version valid where it is written, on the side a node doesn't read | Refused, and reported apart; nothing is held against the node that put it | unit | | 46 | A party's `true` and `false` on one proposal, both in a history | Both count there: a history holding more votes takes back no pass and no death | unit | | 47 | A path whose `{party}` segment names no party (`names/r/foo`) | The `{party}` rule doesn't match; a later rule decides | unit | -| 48 | Two thirds back a proposal that is no version | It can't be written, nor die; the federation is bound to its version. A node shows that it is none, and clients don't back it | unit | +| 48 | Two thirds back a proposal that is no version | It can't die; nodes stop at a version event writing it, until their operators have them drop it (39); the federation is bound to its version. A node shows that it is none, and clients don't back it | unit | | 49 | Keys voting twice part honest nodes | An operator has its node drop what it took and take the other side's; no fork. Where the version it took named a checkpoint, by a checkpoint its operator pins | sim | | 50 | An honest party backs a second proposal once its first is dead; another party later signs `true` on the first too | It isn't named: its first proposal is dead by all the votes; the party with `true` and `false` on one is | unit | | 51 | A node whose operator had it drop a version is sent it again, written anew by another node, or holds the votes that passed it | It refuses any event writing that version and writes it no more, until its operator says otherwise | unit | @@ -805,14 +812,21 @@ and the Automerge version, whose encoding fixes a change's hash. A change to the federation is a design like this one, with the next number. A swarm moves to it by a version of its policy that names it (`"federation": 2`), which the parties pass once their nodes run a continuum that knows it. A node that doesn't know -the federation, or a version, its swarm has stops taking events and says so. +the federation its swarm's root names holds no store, and says so. One that doesn't read +a version that passed on the version it reads stops at the version event writing it, and +says so: it takes nothing and refuses nothing from then on, and keeps none of it across a +restart, until it runs a continuum that reads it, or its operator has it drop that event +([`store.md`](store.md), [A node's own](store.md#a-nodes-own)). Dropped, the node refuses +that event and any writing that version, as any version its operator had it drop, and +goes on under the version before. One beside the version it reads, it refuses of its own, +as any other ([Changing the policy](#changing-the-policy)). A change that only has the federation take what it refused, and reads nothing it took otherwise, keeps the number: the keys of type `ecdsa-sha2-nistp256` taken beside Ed25519 ones are such a change. It is rolled out first and used after: every node runs a continuum that takes it before any party uses it. A node that doesn't refuses what uses -it, a version naming such a key among them, and reads otherwise than the others until -it does. +it, and reads otherwise than the others until it does; at a version naming such a key, +which it doesn't read, it stops, as above. ## The registry on it diff --git a/docs/guides/federation/proposing-and-voting.md b/docs/guides/federation/proposing-and-voting.md index e656083..12f8dfd 100644 --- a/docs/guides/federation/proposing-and-voting.md +++ b/docs/guides/federation/proposing-and-voting.md @@ -42,7 +42,10 @@ Party names are 1 to 64 of `A-Z a-z 0-9 . _ @ -`; a party has up to 16 keys and nodes; a rule's path is a path as a write's is, its first segment literal and not `_continuum`, `**` last only, `{party}` once at most, and its writer a party of the version, `"{party}"` where the path holds it, or `false`. A version breaking any of that -is none: it can't pass, and a party's client backs none. +is none, and a party's client backs none; where two thirds pass it all the same, nodes +stop at the version event writing it, until their operators have them drop it +([`when-things-go-wrong.md`](when-things-go-wrong.md), +[A node stopped](when-things-go-wrong.md#a-node-stopped)). ## Proposing diff --git a/docs/guides/federation/refusals-and-drops.md b/docs/guides/federation/refusals-and-drops.md index b32a819..1d29df1 100644 --- a/docs/guides/federation/refusals-and-drops.md +++ b/docs/guides/federation/refusals-and-drops.md @@ -40,8 +40,9 @@ file`). ## Dropping -An operator may have the node drop events it took after its checkpoint, with everything -built on them: +An operator may have the node drop events it took after its checkpoint, or the version +event it stopped at ([`when-things-go-wrong.md`](when-things-go-wrong.md), +[A node stopped](when-things-go-wrong.md#a-node-stopped)), with everything built on them: ```sh curl -u tv:secret -X POST -H 'Content-Type: application/json' \ @@ -50,7 +51,8 @@ curl -u tv:secret -X POST -H 'Content-Type: application/json' \ ``` It answers what it dropped, in the order taken: those named, version events naming the -version one of them names, and what is built on them. From then on the node refuses those +version one of them names, and what is built on them; then the event it stopped at, if +named, after which it takes again. From then on the node refuses those events from whoever brings them, and any version event naming a version among them, whoever wrote it anew; it writes none itself, though it holds the votes passing it, and keeps so across restarts. Not what its checkpoint keeps diff --git a/docs/guides/federation/upgrades.md b/docs/guides/federation/upgrades.md index 570de19..f91143f 100644 --- a/docs/guides/federation/upgrades.md +++ b/docs/guides/federation/upgrades.md @@ -26,8 +26,11 @@ the snapshot, and the Automerge version, whose encoding fixes a change's hash. T bytes are fixed by vectors that the tests check (`federation::vectors`). There is one federation, 1. A node knows it alone: a version naming another is none to -it, refused where a version event names it, with everything built on it. A proposal -naming another is taken as a proposal, and a party's client backs none. +it. Where one passes, the node stops at the version event writing it, taking nothing and +refusing nothing, until it runs a continuum that knows it, or its operator has it drop +that event ([`when-things-go-wrong.md`](when-things-go-wrong.md), +[A node stopped](when-things-go-wrong.md#a-node-stopped)). A proposal naming another is +taken as a proposal, and a party's client backs none. A change to any of that is a federation of its own, with the next number, and its own design. A swarm moves to it by a version naming it, which the parties pass once their @@ -37,5 +40,5 @@ A change that only has the federation take what it refused, and reads nothing it otherwise, keeps the number: `ecdsa-sha2-nistp256` keys beside Ed25519 ones. Roll it out first, then use it: every party's operators upgrade their nodes, and only then does a party name such a key in a version or a key set, or sign with one. A node not upgraded -refuses what uses it and reads otherwise than the others: a version naming such a key is -none to it, and it stays on the version before. +refuses what uses it and reads otherwise than the others; at a version naming such a key, +none to it, it stops, until it is upgraded. diff --git a/docs/guides/federation/watching.md b/docs/guides/federation/watching.md index f55005a..ec7ce6a 100644 --- a/docs/guides/federation/watching.md +++ b/docs/guides/federation/watching.md @@ -10,6 +10,11 @@ store, through its API and its log. * **A version the node refused** (`own` in `GET /events/refused` not empty): keys voted twice, and nodes may be apart. Act on it at once ([`when-things-go-wrong.md`](when-things-go-wrong.md)). +* **A stop** (`stopped` in `GET /events` not `null`): a version passed that the node's + continuum doesn't read, and it takes nothing until upgraded, or until its operator has + it drop that version event + ([`when-things-go-wrong.md`](when-things-go-wrong.md), + [A node stopped](when-things-go-wrong.md#a-node-stopped)). * **No store** (`404` from `GET /events` for longer than a start takes): the node can't read its store ([`when-things-go-wrong.md`](when-things-go-wrong.md), [A node doing less than it should](when-things-go-wrong.md#a-node-doing-less-than-it-should)). @@ -37,6 +42,9 @@ At warn or error: * `Refused an event its history takes, by what this node took before: kept so`: a version the node refused, as above; +* `The store stopped at an event this continuum can't judge: it takes none until it runs + one that does, or its operator has it drop that one`: a stop, as above; `The store + takes events again` once its operator had it drop that event; * `This node isn't a member of its store, as it reads now: its party doesn't list it, and members don't sync with it`; * `The pinned checkpoint isn't in the document (yet): holding no store until it is`; diff --git a/docs/guides/federation/when-things-go-wrong.md b/docs/guides/federation/when-things-go-wrong.md index c79f24f..8c46e1d 100644 --- a/docs/guides/federation/when-things-go-wrong.md +++ b/docs/guides/federation/when-things-go-wrong.md @@ -1,8 +1,8 @@ # When things go wrong -What to do when a node reports a version it refused, when a party voted twice, and when -a node doesn't do what it should. The rules are the design's -([`federation.md`](../../federation.md), +What to do when a node reports a version it refused, when a party voted twice, when a +node stopped at a version it doesn't read, and when a node doesn't do what it should. The +rules are the design's ([`federation.md`](../../federation.md), [Changing the policy](../../federation.md#changing-the-policy); [`store.md`](../../store.md), [A node's own](../../store.md#a-nodes-own)). @@ -65,6 +65,34 @@ or with its keys named anew ([`forking-and-following.md`](forking-and-following. Tell the other parties what your node took and what it refused, and send them your record. +## A node stopped + +A node stops at a version event writing a version that passed on the version it reads, +which its continuum doesn't read: of a later federation, using what a later continuum +takes, or none at all. It takes nothing and refuses nothing from then on, and votes +nothing by itself or by hand; what is put through it, it still writes into the document +(`stopped` when put). Its log says so (`The store stopped at an event this continuum +can't judge`), and `GET /events` names the event, why, and how many it holds back: + +```sh +curl -s -u tv:secret localhost:3000/events | jq .stopped +# {"heldBack":12,"id":"9a1e…","why":"version 3c0f… passed, and doesn't read as one…"} +``` + +Upgrade the node to a continuum that reads it: restarted, it reads its store anew and +takes it. Where none does (two thirds passed a proposal that is no version), have the node +drop that event: it goes on under the version before, refuses the event and any writing +that version, and keeps so (`--store-file`): + +```sh +curl -u tv:secret -X POST -H 'Content-Type: application/json' \ + --data '{"ids": ["<the event it stopped at>"]}' localhost:3000/admin/events/drop +# {"dropped":["9a1e…"]} +``` + +Allowed again (`/admin/events/allow`), it stops there once more. A node that stopped +keeps none of it in its store file: restarted, it stops there again. + ## A node doing less than it should * **It holds no store** (`404` from `GET /events`): it hasn't read its document yet, its diff --git a/docs/guides/federation/writing-and-reading.md b/docs/guides/federation/writing-and-reading.md index 3dd0578..057aad7 100644 --- a/docs/guides/federation/writing-and-reading.md +++ b/docs/guides/federation/writing-and-reading.md @@ -33,7 +33,10 @@ object, the whole event up to 256 KiB. `null` empties the place. The client signs the write with its key (`--key`), on the heads its node shows, which are what the party read; records it (`--record`), to sign it again after a move; and puts it through the node. The node answers what became of it: `taken`, `held` (taken before), -or `waiting` for an event it builds on that the node doesn't hold yet. +`waiting` for an event it builds on that the node doesn't hold yet, or `stopped`: the +node stopped at a version it doesn't read, and holds it back +([`when-things-go-wrong.md`](when-things-go-wrong.md), +[A node stopped](when-things-go-wrong.md#a-node-stopped)). A write counts while the key that signed it is among its party's keys: those of its key set, or, while it has written none, those its version names diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 3dbcd62..e1e602a 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -1419,9 +1419,12 @@ paths: description: | The store's ID (its root's), its checkpoint's ID, its heads (what a new event builds on) and how many events it took after its checkpoint, as this node reads - it now; the party this node is of (`party`, `--store-party`), and whether it - votes for it by itself (`votesByItself`), and so casts votes by hand - (`POST /admin/events/vote`). + it now; the event it stopped at, if any (`stopped`: a version event writing a + version that passed, which this continuum doesn't read), why, and how many + events it holds back since, that one among them, taking nothing and refusing + nothing until upgraded or told to drop it (`/admin/events/drop`); the party this + node is of (`party`, `--store-party`), and whether it votes for it by itself + (`votesByItself`), and so casts votes by hand (`POST /admin/events/vote`). responses: '200': description: The store. @@ -1429,7 +1432,7 @@ paths: application/json: schema: type: object - required: [store, checkpoint, heads, taken, party, votesByItself] + required: [store, checkpoint, heads, taken, stopped, party, votesByItself] properties: store: type: string @@ -1441,6 +1444,16 @@ paths: type: string taken: type: integer + stopped: + type: [object, 'null'] + required: [id, why, heldBack] + properties: + id: + type: string + why: + type: string + heldBack: + type: integer party: type: [string, 'null'] votesByItself: @@ -1454,7 +1467,8 @@ paths: Puts an event, the body as its bytes, whoever signed it: offered to the store, and written into the document at its ID (the SHA-256 of its bytes) unless the store refuses it. `event` says what became of it: `taken`, `held` (taken - before), or `waiting` for what it builds on. + before), `waiting` for what it builds on, or `stopped`: held back, the store + stopped at a version it doesn't read (`stopped` in `GET /events`). requestBody: required: true content: @@ -1474,7 +1488,7 @@ paths: type: string event: type: string - enum: [taken, held, waiting] + enum: [taken, held, waiting, stopped] '404': $ref: '#/components/responses/Error' '422': @@ -1680,12 +1694,14 @@ paths: tags: [Events] summary: Drop events this node took description: | - Drops events the store took after its checkpoint, version events naming the - version one of them names, and everything built on them: this node refuses them - from then on, from whoever brings them, and any version event naming a version - among them, and keeps so across restarts (`--store-file`), until it is told - otherwise (`/admin/events/allow`). Not what the checkpoint keeps or holds by - ID. Answers what it dropped, in the order taken. Needs `write` on `_continuum`. + Drops events the store took after its checkpoint, or the version event it + stopped at (`stopped` in `GET /events`), version events naming the version one + of them names, and everything built on them: this node refuses them from then + on, from whoever brings them, and any version event naming a version among + them, and keeps so across restarts (`--store-file`), until it is told otherwise + (`/admin/events/allow`). Not what the checkpoint keeps or holds by ID. Answers + what it dropped, in the order taken, then the event it stopped at, after which + it takes again. Needs `write` on `_continuum`. requestBody: required: true content: diff --git a/docs/store.md b/docs/store.md index 3192389..b598539 100644 --- a/docs/store.md +++ b/docs/store.md @@ -114,6 +114,8 @@ A fold says, of its events: its group ([Members](#members)); * where it needs to, **what a node refuses that its history takes**, by what the node took before ([A node's own](#a-nodes-own)); +* where it needs to, **what a node can't judge** that it would take: an event only a fold + knowing more reads, where the node stops ([A node's own](#a-nodes-own)); * where it needs to, **what an event writes that others may write alike**, so that a node dropping it refuses them all (the federation: a version event's version). @@ -126,10 +128,16 @@ pub trait Fold { const LARGEST: usize; /// What the event builds on; `None` where its bytes are none of this fold's events. fn dependencies(event: &[u8]) -> Option<Vec<Id>>; - /// Whether the event is taken where its history reads as `at`. + /// Whether the event is taken where its history reads as `at`, as far as it reads it. fn judge(at: &Self::State, id: &Id, event: &[u8]) -> Result<(), Refused>; /// Adds the event, whose dependencies are added already. fn add(state: &mut Self::State, id: &Id, event: &[u8]); + /// Why a node whose store reads as `now` can't judge an event its history doesn't + /// refuse, that it would take, where it can't: the store stops at it. + fn stops(_at: &Self::State, _now: &Self::State, _id: &Id, _event: &[u8]) + -> Option<String> { + None + } /// Whether a node whose store reads as `now` takes an event its history takes. fn keeps(_now: &Self::State, _id: &Id, _event: &[u8]) -> bool { true @@ -169,10 +177,10 @@ which its tests check ([Tests](#tests)). ## What is promised 1. **Every node takes an event or none**, but for what a fold has a node refuse by what - it took, what an operator has its node drop ([A node's own](#a-nodes-own)), and what - moving to a checkpoint refuses. An event is judged at its history, which its ID fixes, - so every node judges it alike, whenever and from whomever it comes. An event refused is - refused with everything built on it. + it took, what an operator has its node drop, what a node stops at + ([A node's own](#a-nodes-own)), and what moving to a checkpoint refuses. An event is + judged at its history, which its ID fixes, so every node judges it alike, whenever and + from whomever it comes. An event refused is refused with everything built on it. 2. **Nodes that hold the same checkpoint and the same events after it read the same state**, whatever order the events came in, and whatever carried them. 3. **Nothing underneath adds an event or takes one away.** What a node holds is what it @@ -203,6 +211,16 @@ so, and its operator's drops, in a file of its own, across restarts. No peer's word moves a node; its operator's does. A node that refuses what another took holds other events, and reads its own state from them; two that hold the same read the same. +A fold may have a node **stop** at an event it can't judge, that it would take: one only +a fold knowing more reads (the federation's node: a version event writing a version +that passed, which its continuum doesn't read, +[Changing the federation](federation.md#changing-the-federation)). The node takes +nothing and refuses nothing from then on: it holds back whatever comes, that event first, +keeps none of it across a restart, where it stops there again, and reports it, until it +runs a continuum that reads the event, or its operator has it drop the event, or one it +builds on. Then it refuses it, as dropped or as built on what was, with everything built +on it, and judges what it held back. What it took before it stopped, it holds as it did. + ## Forks A **fork** is a store whose root is a checkpoint at heads of another, which its maker @@ -335,7 +353,9 @@ coordinator): two remakes at one base are two epochs. checkpoint forgot, and events nobody takes, leave the document here. A node that moved to a later checkpoint since the base builds the snapshot as one that didn't, as both read the document; nodes that read the store differently at the base (the sides of - keys that voted twice, in the federation) build different snapshots, and part here. + keys that voted twice, in the federation, or a node that stopped, holding back what + the others took: [A node's own](#a-nodes-own)) build different snapshots, and part + here. * **A node follows a decision** whichever node coordinated it: a decision reaches it only through a member, which could decide one itself, and a remake changes no event. It builds the snapshot itself, from a base it holds, and checks it against the |
