diff options
| -rw-r--r-- | docs/federation.md | 7 | ||||
| -rw-r--r-- | docs/guides/federation/upgrades.md | 43 | ||||
| -rw-r--r-- | docs/guides/federation/watching.md | 4 | ||||
| -rw-r--r-- | docs/guides/federation/writing-and-reading.md | 5 | ||||
| -rw-r--r-- | docs/openapi.yaml | 39 | ||||
| -rw-r--r-- | docs/peers.md | 7 | ||||
| -rw-r--r-- | src/api/events.rs | 46 | ||||
| -rw-r--r-- | src/federation/client.rs | 66 | ||||
| -rw-r--r-- | src/federation/party.rs | 42 | ||||
| -rw-r--r-- | src/swarm/speaking.rs | 26 |
10 files changed, 260 insertions, 25 deletions
diff --git a/docs/federation.md b/docs/federation.md index 8dc23b3..615bc05 100644 --- a/docs/federation.md +++ b/docs/federation.md @@ -904,8 +904,11 @@ swarm's root names holds no store, and says so. So a revision is rolled out, then passed: every party's operators restart its nodes on a continuum that knows it, and the parties pass a version naming it once theirs do, as a -party's client backs no version it doesn't read. Who hasn't by then stops at that version, -and takes part again once it has. Once a store moved to a checkpoint at a version naming +party's client backs no version its node doesn't read. Each node tells its peers the +newest revision it knows (SPEAKS, [`peers.md`](peers.md), +[Handshake and join](peers.md#handshake-and-join)), and a party's client warns, backing +one, of every node a party lists not known to know its revision. Who hasn't upgraded by +then stops at that version, and takes part again once it has. Once a store moved to a checkpoint at a version naming a revision, nodes join it only from a pinned checkpoint, and nobody judges what came before again: a later continuum may hold only stores at that revision or after, and leave out the rules of the revisions before. diff --git a/docs/guides/federation/upgrades.md b/docs/guides/federation/upgrades.md index 0a1a094..55dd921 100644 --- a/docs/guides/federation/upgrades.md +++ b/docs/guides/federation/upgrades.md @@ -15,7 +15,9 @@ the same Automerge on every node, and one with another rejoins instead of switch continuum's first release, take every upgrade as such a change. Nobody runs every node of a federated swarm: each party's operators upgrade its nodes. -Agree on when, among the parties, and upgrade close together. +Agree on when, among the parties, and upgrade close together. Which wire versions each +node the parties list speaks, as far as a node heard from it, shows in +`GET /events/counts` (`knows`, below: `wire`, the oldest and the newest). ## The federation's number and its revision @@ -61,7 +63,22 @@ adds is refused everywhere until a version names it. ``` and the revision of the current version, in `GET /events/counts` - (`.version.revision.named`). + (`.version.revision.named`). There too, of each node the parties list, the newest + revision it said it knows (`knows`; [`peers.md`](../../peers.md), + [Handshake and join](../../peers.md#handshake-and-join)), as far as this node heard + from it: the nodes still to upgrade, every party's. + + ```sh + curl -s -u tv:secret localhost:3000/events/counts | jq -r '.parties | to_entries[] + | .key as $party | .value.knows | to_entries[] + | "\($party) \(.key): \(.value.revision.named // "said nothing")"' + # lass continuum:9c07…: the base revision + # tv continuum:5be1…: revision 1 (merge-events) + # tv continuum:77d3…: said nothing + ``` + + A node says nothing on a build from before it said this, or where this node didn't + hear from it since it started: each says it with each handshake. 2. **Propose.** A party begins the next version following it, by the feature it adds; `next` says so where its node knows a later revision than the current version's: @@ -74,8 +91,9 @@ adds is refused everywhere until a version names it. # proposal 5e81… ``` - It may already hold what the revision adds to a version: a version is read by the - revision it names. Its proposal and the votes on it are judged under the current + Where nodes the parties list aren't known to know the revision, `next` warns, and + names them. It may already hold what the revision adds to a version: a version is read + by the revision it names. Its proposal and the votes on it are judged under the current version, and what the revision adds to events counts from the version after. 3. **Vote.** Each party backs it once every node of the party knows the revision: @@ -90,10 +108,19 @@ adds is refused everywhere until a version names it. --record tv.record vote 5e81… yes ``` - The client backs it only where its node knows the revision, and says so otherwise; it - can't see the party's other nodes, which its operators check. A node voting by itself - backs no version changing the revision: vote on it by hand, through that node. A party - not ready leaves it be, or votes `false` to end it. + The client backs it only where its node knows the revision, and says so otherwise. It + backs it where other nodes, of any party, aren't known to know it, and warns, naming + them and what each said (`knows`, above): once it passes, those stop at it till + upgraded. + + ``` + Warning: these nodes aren't known to know revision 1 (merge-events), as far as the + node heard from them; once a version of it passes, each stops at it till upgraded: + tv: continuum:77d3… said nothing: a build from before SPEAKS, or not heard from + ``` + + A node voting by itself backs no version changing the revision: vote on it by hand, + through that node. A party not ready leaves it be, or votes `false` to end it. 4. **After.** Once two thirds passed it, what the revision adds counts from that version on. A node that doesn't know it stops at that version (`stopped` in `GET /events`, diff --git a/docs/guides/federation/watching.md b/docs/guides/federation/watching.md index da4a93f..9db3399 100644 --- a/docs/guides/federation/watching.md +++ b/docs/guides/federation/watching.md @@ -25,6 +25,10 @@ store, through its API and its log. is worth a look at what passed and who voted for it. * **The members** (`members` in `GET /events/counts`): a node of your party missing there syncs with nobody. +* **What the nodes run** (`knows` of each party in `GET /events/counts`): the wire + versions each node speaks and the newest revision it knows, as far as the node heard + from it: the nodes still to upgrade before a revision passes + ([`upgrades.md`](upgrades.md), [Moving to a revision](upgrades.md#moving-to-a-revision)). * **Events beyond the checkpoint** (`taken` in `GET /events`): what a move to a checkpoint would let nodes forget; a node voting by itself proposes one at a count you set ([`voting-by-itself.md`](voting-by-itself.md)). diff --git a/docs/guides/federation/writing-and-reading.md b/docs/guides/federation/writing-and-reading.md index 057aad7..d45bf0d 100644 --- a/docs/guides/federation/writing-and-reading.md +++ b/docs/guides/federation/writing-and-reading.md @@ -97,7 +97,10 @@ Votes and proposals aren't signed again: their version is behind. `GET /events/counts` is what counts, as the node reads it now: * `version`: the current version, its `number`, its `hash`, and the version itself; -* `parties`: each party's `keys` and `nodes`, as they count; +* `parties`: each party's `keys` and `nodes`, as they count, and of those nodes what + each said it speaks and knows (`knows`: the wire versions, the newest revision; null + where the node heard nothing from it), for upgrades + ([`upgrades.md`](upgrades.md), [Moving to a revision](upgrades.md#moving-to-a-revision)); * `values`: each place holding a value, with it; * `members`: each node of the swarm, with its party, `[node, party]`; * `proposals`: the proposals on the current version, by hash, each with the events diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 05e29d5..b1e1e28 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -1550,7 +1550,12 @@ paths: description: | What counts, as the store reads now (docs/federation.md): the current version (its number, its hash, its revision, by number and ID and as people read it, and - the version itself), each party's keys and nodes, the + the version itself), each party's keys and nodes, what each of those nodes said + it speaks and knows (`knows`, by node ID: the wire versions it speaks, the oldest + and the newest, and the newest revision of the federation it knows, by number + and ID and as people read it, its number null where this node doesn't know it; + null where the node said nothing, a build from before SPEAKS or one this node + didn't hear from; this node's own as it is), the value at each place, the members, each `[node, party]`, and the proposals on the current version, by their hash, each with the IDs of the events proposing it that a vote may build on (`proposals`: of a version naming a checkpoint, those above @@ -1576,6 +1581,38 @@ paths: type: object parties: type: object + additionalProperties: + type: object + required: [keys, nodes, knows] + properties: + keys: + type: array + items: + type: string + nodes: + type: array + items: + type: string + knows: + type: object + additionalProperties: + type: [object, 'null'] + required: [wire, revision] + properties: + wire: + type: array + items: + type: integer + revision: + type: [object, 'null'] + required: [number, id, named] + properties: + number: + type: [integer, 'null'] + id: + type: string + named: + type: string values: type: object members: diff --git a/docs/peers.md b/docs/peers.md index c63bebe..d2f822f 100644 --- a/docs/peers.md +++ b/docs/peers.md @@ -99,9 +99,10 @@ Before each HELLO and each HELLO_REPLY, a node sends SPEAKS: the wire versions i and the newest revision of the federation it knows, as its build has them. It stands outside the wire versions, laid out as no build changes, so that a peer on another build reads it too, once that peer reads the node's WHO and ME -([Node IDs and addresses](#node-ids-and-addresses)). A build from before SPEAKS takes it -for a message of another wire version, and logs that once of the peer -([`operations.md`](operations.md), [Upgrading](operations.md#upgrading)). +([Node IDs and addresses](#node-ids-and-addresses)): what each peer said shows in `knows` +of `GET /events/counts`, for the nodes a federated store's parties list. A build from +before SPEAKS takes it for a message of another wire version, and logs that once of the +peer ([`operations.md`](operations.md), [Upgrading](operations.md#upgrading)). Every HELLO carries a random id of the sender's current run, and the HELLO_REPLY echoes it. A reply with another id answers a HELLO from an earlier run (delivered late, or diff --git a/src/api/events.rs b/src/api/events.rs index 7bb322e..f49c540 100644 --- a/src/api/events.rs +++ b/src/api/events.rs @@ -106,6 +106,41 @@ async fn summary(State(state): State<AppState>) -> Response { }) } +/// What `node` says it speaks and knows (SPEAKS, `swarm::speaking`), this node's own as +/// it is: the wire versions it speaks, the oldest and the newest, and the newest revision +/// of the federation it knows, by number and ID and as people read it, its number null +/// where this continuum doesn't know it. Null where the node said nothing: a build from +/// before SPEAKS, or one not heard from. +fn knows(state: &AppState, node: &str) -> JsonValue { + let speaks = if node == state.local_id() { + Some(crate::swarm::speaking::own()) + } else { + state.speaking.of(node) + }; + let Some(speaks) = speaks else { + return JsonValue::Null; + }; + let revision = speaks.revision.map(|id| { + let id = Id(id); + match Revision::of(&id) { + Some(revision) => json!({ + "number": revision.0, + "id": id.to_string(), + "named": revision.to_string(), + }), + None => json!({ + "number": null, + "id": id.to_string(), + "named": format!("{id}, a revision this continuum doesn't know"), + }), + } + }); + json!({ + "wire": [speaks.oldest, speaks.newest], + "revision": revision, + }) +} + /// The revisions this continuum knows, the base revision first: each its number, its ID, /// the feature it adds and what that adds, and as people read it. fn revisions() -> Vec<JsonValue> { @@ -143,7 +178,8 @@ async fn event(State(state): State<AppState>, Path(id): Path<String>) -> Respons } /// `GET /events/counts`: what counts, as the store reads now: the current version, each -/// party's keys and nodes, the values at places, the members, and the proposals on the +/// party's keys and nodes, what each of those nodes says it speaks and knows (`knows`), +/// the values at places, the members, and the proposals on the /// current version, each with the events proposing it a vote may build on (`proposing`: /// for a checkpoint, those above its heads), the votes on it, the parties /// voting `true` and `false`, whether it passed and whether it is dead, and why a party's @@ -157,11 +193,17 @@ async fn counts(State(state): State<AppState>) -> Response { .parties .keys() .map(|party| { + let nodes = read.nodes_of(party, current); + let knows: serde_json::Map<String, JsonValue> = nodes + .iter() + .map(|node| (node.clone(), knows(&state, node))) + .collect(); ( party.clone(), json!({ "keys": read.keys_of(party, current), - "nodes": read.nodes_of(party, current), + "nodes": nodes, + "knows": knows, }), ) }) diff --git a/src/federation/client.rs b/src/federation/client.rs index 7b9d3d7..45f5019 100644 --- a/src/federation/client.rs +++ b/src/federation/client.rs @@ -22,6 +22,7 @@ use serde::{Deserialize, Serialize}; use super::event::{Does, Event, Signed}; use super::jcs::{self, Json}; +use super::revision::Revision; use super::store::Id; use super::version::Version; @@ -480,6 +481,41 @@ pub(crate) fn changes(current: &Json, proposed: &Json) -> Vec<String> { lines } +/// Of the parties' nodes, those not known to know `revision`, by what the node a client +/// talks to heard each say it knows (`knows` in `counts`, from `GET /events/counts`): a +/// line for each party with any, naming them and what each said. Once a version of +/// `revision` passes, such a node stops at it till upgraded. None where every node said +/// it knows it. +pub(crate) fn unready(counts: &serde_json::Value, revision: Revision) -> Vec<String> { + let mut lines = Vec::new(); + for (party, of) in counts["parties"].as_object().into_iter().flatten() { + let behind: Vec<String> = of["knows"] + .as_object() + .into_iter() + .flatten() + .filter_map(|(node, knows)| { + let newest = &knows["revision"]; + let named = newest["named"].as_str().unwrap_or("?"); + match newest["number"].as_u64() { + Some(number) if usize::try_from(number).is_ok_and(|n| n >= revision.0) => None, + Some(_) => Some(format!("{node} knows {named} at newest")), + None if newest.is_object() => Some(format!("{node} knows {named}")), + None if knows.is_object() => Some(format!( + "{node} knows no revision: built without the federation" + )), + None => Some(format!( + "{node} said nothing: a build from before SPEAKS, or not heard from" + )), + } + }) + .collect(); + if !behind.is_empty() { + lines.push(format!("{party}: {}", behind.join("; "))); + } + } + lines +} + /// The parties that voted twice, by the votes `records` keep: of each version they name, /// a party whose `true` and `false` are on one proposal, or whose `true` is on two /// proposals of which neither is dead by all the votes. Each vote counts as checked from @@ -908,4 +944,34 @@ mod tests { )] ); } + + /// Of the nodes `knows` names, those that didn't say they know a revision are named, + /// with what each said; a party whose nodes all did, not. + #[test] + fn nodes_not_known_to_know_a_revision_are_named() { + let one = Revision(1); + let id = one.id().to_string(); + let counts = serde_json::json!({"parties": { + "lass": {"knows": { + "a": {"wire": [52, 52], "revision": {"number": 1, "id": id, "named": "one"}}, + }}, + "tv": {"knows": { + "b": {"wire": [52, 52], "revision": {"number": 0, "named": "the base"}}, + "c": null, + "d": {"wire": [52, 52], "revision": {"number": null, "named": "x, unknown"}}, + "e": {"wire": [52, 52], "revision": null}, + "f": {"wire": [52, 53], "revision": {"number": 2, "named": "two"}}, + }}, + }}); + assert_eq!( + unready(&counts, one), + [ + "tv: b knows the base at newest; c said nothing: a build from before SPEAKS, \ + or not heard from; d knows x, unknown; e knows no revision: built without \ + the federation" + ] + ); + assert!(unready(&counts, Revision(0)).len() == 1); + assert!(unready(&serde_json::json!({}), one).is_empty()); + } } diff --git a/src/federation/party.rs b/src/federation/party.rs index a12dc7f..2c8dc4b 100644 --- a/src/federation/party.rs +++ b/src/federation/party.rs @@ -314,7 +314,15 @@ pub(crate) async fn main() -> Result<()> { moves to it, once every party's nodes run a continuum that knows it", current.revision ); + warn_unready(&node.counts().await?, later); } + } else if let Some(revision) = next["revision"] + .as_str() + .and_then(Id::from_hex) + .and_then(|id| Revision::of(&id)) + && revision != current.revision + { + warn_unready(&node.counts().await?, revision); } println!("{}", serde_json::to_string_pretty(&next)?); return Ok(()); @@ -755,18 +763,21 @@ async fn vote( })) if hash == proposal => version, _ => bail!("the node's {proposing} isn't the proposal it names"), }; - // A later revision: the node knows it, or it stops at the version once it passes. + // A later revision: the node knows it, or it stops at the version once it passes; and + // so does every other node not known to know it. if yes && let Ok(version) = Version::read(&proposed) && version.revision != current.revision - && !node.revisions().await?.contains(&version.revision.id()) { - bail!( - "{proposal} moves to {}, which the node doesn't know: once it passed, a node that \ - doesn't stops at it, and votes nothing till upgraded; upgrade this node and every \ - other of the party to a continuum that knows it, then back it", - version.revision - ); + if !node.revisions().await?.contains(&version.revision.id()) { + bail!( + "{proposal} moves to {}, which the node doesn't know: once it passed, a node \ + that doesn't stops at it, and votes nothing till upgraded; upgrade this node \ + and every other of the party to a continuum that knows it, then back it", + version.revision + ); + } + warn_unready(&counts, version.revision); } // The votes on its party's earlier proposals it backed, for the nays among them. let mut nays = Vec::new(); @@ -816,6 +827,21 @@ async fn vote( node.put(&cast).await } +/// Warns of the nodes not known to know `revision`, by `counts` (`client::unready`). +fn warn_unready(counts: &JsonValue, revision: Revision) { + let behind = client::unready(counts, revision); + if behind.is_empty() { + return; + } + eprintln!( + "Warning: these nodes aren't known to know {revision}, as far as the node heard from \ + them; once a version of it passes, each stops at it till upgraded:" + ); + for line in behind { + eprintln!(" {line}"); + } +} + /// Signs again what a move refused: of the latest event it signed at each place, its key /// set and its node list, those the node refuses as not above its checkpoint's heads, or /// as built on one it refuses. diff --git a/src/swarm/speaking.rs b/src/swarm/speaking.rs index dcbfac4..e1a8d2c 100644 --- a/src/swarm/speaking.rs +++ b/src/swarm/speaking.rs @@ -43,4 +43,30 @@ impl Speaking { } heard.insert(peer.to_owned(), speaks); } + + /// What `peer` said last, if it said anything. + pub(crate) fn of(&self, peer: &str) -> Option<Speaks> { + self.0.lock_or_recover().get(peer).copied() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_peer_is_heard_as_it_spoke_last() { + let speaking = Speaking::default(); + assert_eq!(speaking.of("a"), None); + speaking.heard("a", own()); + let older = Speaks { + oldest: 1, + newest: 1, + revision: None, + }; + speaking.heard("b", older); + speaking.heard("a", older); + assert_eq!(speaking.of("a"), Some(older)); + assert_eq!(own().newest, wire::VERSION); + } } |
