aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authortv <tv@krebsco.de>2026-11-19 13:34:24 +0000
committertv <tv@krebsco.de>2026-11-19 14:39:54 +0100
commit9164ebfe9a4014cbbaaaff90c5a1d722e39457fd (patch)
tree9a45156bd4919054939d12c97a19672e92e07b98
parent7a5aeb02024ddff71ba60f40913555cd21a633ad (diff)
federation: which nodes know a revision, in counts and before a `true`
`GET /events/counts` says, of each node a party lists, what it said it speaks and knows (`knows`, by node ID): the wire versions, the oldest and the newest, and the newest revision of the federation it knows, by number and ID and as people read it, its number null where this continuum doesn't know it; null where it said nothing, a build from before SPEAKS or one the node didn't hear from; the node's own as it is. So the nodes still to upgrade before a revision passes show, every party's, where before a party's operators could check only their own. A party's client, backing a proposal that moves to another revision, warns of every node of any party not known to know it, a line a party naming each and what it said: once it passes, those stop at it till upgraded. `next` warns alike, of the revision it moves to, or of the later one it suggests. A warning, not a refusal: what a node heard is no proof, and a node not heard from may well be upgraded.
-rw-r--r--docs/federation.md7
-rw-r--r--docs/guides/federation/upgrades.md43
-rw-r--r--docs/guides/federation/watching.md4
-rw-r--r--docs/guides/federation/writing-and-reading.md5
-rw-r--r--docs/openapi.yaml39
-rw-r--r--docs/peers.md7
-rw-r--r--src/api/events.rs46
-rw-r--r--src/federation/client.rs66
-rw-r--r--src/federation/party.rs42
-rw-r--r--src/swarm/speaking.rs26
10 files changed, 260 insertions, 25 deletions
diff --git a/docs/federation.md b/docs/federation.md
index 8dc23b3..615bc05 100644
--- a/docs/federation.md
+++ b/docs/federation.md
@@ -904,8 +904,11 @@ swarm's root names holds no store, and says so.
So a revision is rolled out, then passed: every party's operators restart its nodes on a
continuum that knows it, and the parties pass a version naming it once theirs do, as a
-party's client backs no version it doesn't read. Who hasn't by then stops at that version,
-and takes part again once it has. Once a store moved to a checkpoint at a version naming
+party's client backs no version its node doesn't read. Each node tells its peers the
+newest revision it knows (SPEAKS, [`peers.md`](peers.md),
+[Handshake and join](peers.md#handshake-and-join)), and a party's client warns, backing
+one, of every node a party lists not known to know its revision. Who hasn't upgraded by
+then stops at that version, and takes part again once it has. Once a store moved to a checkpoint at a version naming
a revision, nodes join it only from a pinned checkpoint, and nobody judges what came
before again: a later continuum may hold only stores at that revision or after, and leave
out the rules of the revisions before.
diff --git a/docs/guides/federation/upgrades.md b/docs/guides/federation/upgrades.md
index 0a1a094..55dd921 100644
--- a/docs/guides/federation/upgrades.md
+++ b/docs/guides/federation/upgrades.md
@@ -15,7 +15,9 @@ the same Automerge on every node, and one with another rejoins instead of switch
continuum's first release, take every upgrade as such a change.
Nobody runs every node of a federated swarm: each party's operators upgrade its nodes.
-Agree on when, among the parties, and upgrade close together.
+Agree on when, among the parties, and upgrade close together. Which wire versions each
+node the parties list speaks, as far as a node heard from it, shows in
+`GET /events/counts` (`knows`, below: `wire`, the oldest and the newest).
## The federation's number and its revision
@@ -61,7 +63,22 @@ adds is refused everywhere until a version names it.
```
and the revision of the current version, in `GET /events/counts`
- (`.version.revision.named`).
+ (`.version.revision.named`). There too, of each node the parties list, the newest
+ revision it said it knows (`knows`; [`peers.md`](../../peers.md),
+ [Handshake and join](../../peers.md#handshake-and-join)), as far as this node heard
+ from it: the nodes still to upgrade, every party's.
+
+ ```sh
+ curl -s -u tv:secret localhost:3000/events/counts | jq -r '.parties | to_entries[]
+ | .key as $party | .value.knows | to_entries[]
+ | "\($party) \(.key): \(.value.revision.named // "said nothing")"'
+ # lass continuum:9c07…: the base revision
+ # tv continuum:5be1…: revision 1 (merge-events)
+ # tv continuum:77d3…: said nothing
+ ```
+
+ A node says nothing on a build from before it said this, or where this node didn't
+ hear from it since it started: each says it with each handshake.
2. **Propose.** A party begins the next version following it, by the feature it adds;
`next` says so where its node knows a later revision than the current version's:
@@ -74,8 +91,9 @@ adds is refused everywhere until a version names it.
# proposal 5e81…
```
- It may already hold what the revision adds to a version: a version is read by the
- revision it names. Its proposal and the votes on it are judged under the current
+ Where nodes the parties list aren't known to know the revision, `next` warns, and
+ names them. It may already hold what the revision adds to a version: a version is read
+ by the revision it names. Its proposal and the votes on it are judged under the current
version, and what the revision adds to events counts from the version after.
3. **Vote.** Each party backs it once every node of the party knows the revision:
@@ -90,10 +108,19 @@ adds is refused everywhere until a version names it.
--record tv.record vote 5e81… yes
```
- The client backs it only where its node knows the revision, and says so otherwise; it
- can't see the party's other nodes, which its operators check. A node voting by itself
- backs no version changing the revision: vote on it by hand, through that node. A party
- not ready leaves it be, or votes `false` to end it.
+ The client backs it only where its node knows the revision, and says so otherwise. It
+ backs it where other nodes, of any party, aren't known to know it, and warns, naming
+ them and what each said (`knows`, above): once it passes, those stop at it till
+ upgraded.
+
+ ```
+ Warning: these nodes aren't known to know revision 1 (merge-events), as far as the
+ node heard from them; once a version of it passes, each stops at it till upgraded:
+ tv: continuum:77d3… said nothing: a build from before SPEAKS, or not heard from
+ ```
+
+ A node voting by itself backs no version changing the revision: vote on it by hand,
+ through that node. A party not ready leaves it be, or votes `false` to end it.
4. **After.** Once two thirds passed it, what the revision adds counts from that version
on. A node that doesn't know it stops at that version (`stopped` in `GET /events`,
diff --git a/docs/guides/federation/watching.md b/docs/guides/federation/watching.md
index da4a93f..9db3399 100644
--- a/docs/guides/federation/watching.md
+++ b/docs/guides/federation/watching.md
@@ -25,6 +25,10 @@ store, through its API and its log.
is worth a look at what passed and who voted for it.
* **The members** (`members` in `GET /events/counts`): a node of your party missing
there syncs with nobody.
+* **What the nodes run** (`knows` of each party in `GET /events/counts`): the wire
+ versions each node speaks and the newest revision it knows, as far as the node heard
+ from it: the nodes still to upgrade before a revision passes
+ ([`upgrades.md`](upgrades.md), [Moving to a revision](upgrades.md#moving-to-a-revision)).
* **Events beyond the checkpoint** (`taken` in `GET /events`): what a move to a
checkpoint would let nodes forget; a node voting by itself proposes one at a count you
set ([`voting-by-itself.md`](voting-by-itself.md)).
diff --git a/docs/guides/federation/writing-and-reading.md b/docs/guides/federation/writing-and-reading.md
index 057aad7..d45bf0d 100644
--- a/docs/guides/federation/writing-and-reading.md
+++ b/docs/guides/federation/writing-and-reading.md
@@ -97,7 +97,10 @@ Votes and proposals aren't signed again: their version is behind.
`GET /events/counts` is what counts, as the node reads it now:
* `version`: the current version, its `number`, its `hash`, and the version itself;
-* `parties`: each party's `keys` and `nodes`, as they count;
+* `parties`: each party's `keys` and `nodes`, as they count, and of those nodes what
+ each said it speaks and knows (`knows`: the wire versions, the newest revision; null
+ where the node heard nothing from it), for upgrades
+ ([`upgrades.md`](upgrades.md), [Moving to a revision](upgrades.md#moving-to-a-revision));
* `values`: each place holding a value, with it;
* `members`: each node of the swarm, with its party, `[node, party]`;
* `proposals`: the proposals on the current version, by hash, each with the events
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 05e29d5..b1e1e28 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -1550,7 +1550,12 @@ paths:
description: |
What counts, as the store reads now (docs/federation.md): the current version
(its number, its hash, its revision, by number and ID and as people read it, and
- the version itself), each party's keys and nodes, the
+ the version itself), each party's keys and nodes, what each of those nodes said
+ it speaks and knows (`knows`, by node ID: the wire versions it speaks, the oldest
+ and the newest, and the newest revision of the federation it knows, by number
+ and ID and as people read it, its number null where this node doesn't know it;
+ null where the node said nothing, a build from before SPEAKS or one this node
+ didn't hear from; this node's own as it is), the
value at each place, the members, each `[node, party]`, and the proposals on the
current version, by their hash, each with the IDs of the events proposing it that
a vote may build on (`proposals`: of a version naming a checkpoint, those above
@@ -1576,6 +1581,38 @@ paths:
type: object
parties:
type: object
+ additionalProperties:
+ type: object
+ required: [keys, nodes, knows]
+ properties:
+ keys:
+ type: array
+ items:
+ type: string
+ nodes:
+ type: array
+ items:
+ type: string
+ knows:
+ type: object
+ additionalProperties:
+ type: [object, 'null']
+ required: [wire, revision]
+ properties:
+ wire:
+ type: array
+ items:
+ type: integer
+ revision:
+ type: [object, 'null']
+ required: [number, id, named]
+ properties:
+ number:
+ type: [integer, 'null']
+ id:
+ type: string
+ named:
+ type: string
values:
type: object
members:
diff --git a/docs/peers.md b/docs/peers.md
index c63bebe..d2f822f 100644
--- a/docs/peers.md
+++ b/docs/peers.md
@@ -99,9 +99,10 @@ Before each HELLO and each HELLO_REPLY, a node sends SPEAKS: the wire versions i
and the newest revision of the federation it knows, as its build has them. It stands
outside the wire versions, laid out as no build changes, so that a peer on another build
reads it too, once that peer reads the node's WHO and ME
-([Node IDs and addresses](#node-ids-and-addresses)). A build from before SPEAKS takes it
-for a message of another wire version, and logs that once of the peer
-([`operations.md`](operations.md), [Upgrading](operations.md#upgrading)).
+([Node IDs and addresses](#node-ids-and-addresses)): what each peer said shows in `knows`
+of `GET /events/counts`, for the nodes a federated store's parties list. A build from
+before SPEAKS takes it for a message of another wire version, and logs that once of the
+peer ([`operations.md`](operations.md), [Upgrading](operations.md#upgrading)).
Every HELLO carries a random id of the sender's current run, and the HELLO_REPLY echoes
it. A reply with another id answers a HELLO from an earlier run (delivered late, or
diff --git a/src/api/events.rs b/src/api/events.rs
index 7bb322e..f49c540 100644
--- a/src/api/events.rs
+++ b/src/api/events.rs
@@ -106,6 +106,41 @@ async fn summary(State(state): State<AppState>) -> Response {
})
}
+/// What `node` says it speaks and knows (SPEAKS, `swarm::speaking`), this node's own as
+/// it is: the wire versions it speaks, the oldest and the newest, and the newest revision
+/// of the federation it knows, by number and ID and as people read it, its number null
+/// where this continuum doesn't know it. Null where the node said nothing: a build from
+/// before SPEAKS, or one not heard from.
+fn knows(state: &AppState, node: &str) -> JsonValue {
+ let speaks = if node == state.local_id() {
+ Some(crate::swarm::speaking::own())
+ } else {
+ state.speaking.of(node)
+ };
+ let Some(speaks) = speaks else {
+ return JsonValue::Null;
+ };
+ let revision = speaks.revision.map(|id| {
+ let id = Id(id);
+ match Revision::of(&id) {
+ Some(revision) => json!({
+ "number": revision.0,
+ "id": id.to_string(),
+ "named": revision.to_string(),
+ }),
+ None => json!({
+ "number": null,
+ "id": id.to_string(),
+ "named": format!("{id}, a revision this continuum doesn't know"),
+ }),
+ }
+ });
+ json!({
+ "wire": [speaks.oldest, speaks.newest],
+ "revision": revision,
+ })
+}
+
/// The revisions this continuum knows, the base revision first: each its number, its ID,
/// the feature it adds and what that adds, and as people read it.
fn revisions() -> Vec<JsonValue> {
@@ -143,7 +178,8 @@ async fn event(State(state): State<AppState>, Path(id): Path<String>) -> Respons
}
/// `GET /events/counts`: what counts, as the store reads now: the current version, each
-/// party's keys and nodes, the values at places, the members, and the proposals on the
+/// party's keys and nodes, what each of those nodes says it speaks and knows (`knows`),
+/// the values at places, the members, and the proposals on the
/// current version, each with the events proposing it a vote may build on (`proposing`:
/// for a checkpoint, those above its heads), the votes on it, the parties
/// voting `true` and `false`, whether it passed and whether it is dead, and why a party's
@@ -157,11 +193,17 @@ async fn counts(State(state): State<AppState>) -> Response {
.parties
.keys()
.map(|party| {
+ let nodes = read.nodes_of(party, current);
+ let knows: serde_json::Map<String, JsonValue> = nodes
+ .iter()
+ .map(|node| (node.clone(), knows(&state, node)))
+ .collect();
(
party.clone(),
json!({
"keys": read.keys_of(party, current),
- "nodes": read.nodes_of(party, current),
+ "nodes": nodes,
+ "knows": knows,
}),
)
})
diff --git a/src/federation/client.rs b/src/federation/client.rs
index 7b9d3d7..45f5019 100644
--- a/src/federation/client.rs
+++ b/src/federation/client.rs
@@ -22,6 +22,7 @@ use serde::{Deserialize, Serialize};
use super::event::{Does, Event, Signed};
use super::jcs::{self, Json};
+use super::revision::Revision;
use super::store::Id;
use super::version::Version;
@@ -480,6 +481,41 @@ pub(crate) fn changes(current: &Json, proposed: &Json) -> Vec<String> {
lines
}
+/// Of the parties' nodes, those not known to know `revision`, by what the node a client
+/// talks to heard each say it knows (`knows` in `counts`, from `GET /events/counts`): a
+/// line for each party with any, naming them and what each said. Once a version of
+/// `revision` passes, such a node stops at it till upgraded. None where every node said
+/// it knows it.
+pub(crate) fn unready(counts: &serde_json::Value, revision: Revision) -> Vec<String> {
+ let mut lines = Vec::new();
+ for (party, of) in counts["parties"].as_object().into_iter().flatten() {
+ let behind: Vec<String> = of["knows"]
+ .as_object()
+ .into_iter()
+ .flatten()
+ .filter_map(|(node, knows)| {
+ let newest = &knows["revision"];
+ let named = newest["named"].as_str().unwrap_or("?");
+ match newest["number"].as_u64() {
+ Some(number) if usize::try_from(number).is_ok_and(|n| n >= revision.0) => None,
+ Some(_) => Some(format!("{node} knows {named} at newest")),
+ None if newest.is_object() => Some(format!("{node} knows {named}")),
+ None if knows.is_object() => Some(format!(
+ "{node} knows no revision: built without the federation"
+ )),
+ None => Some(format!(
+ "{node} said nothing: a build from before SPEAKS, or not heard from"
+ )),
+ }
+ })
+ .collect();
+ if !behind.is_empty() {
+ lines.push(format!("{party}: {}", behind.join("; ")));
+ }
+ }
+ lines
+}
+
/// The parties that voted twice, by the votes `records` keep: of each version they name,
/// a party whose `true` and `false` are on one proposal, or whose `true` is on two
/// proposals of which neither is dead by all the votes. Each vote counts as checked from
@@ -908,4 +944,34 @@ mod tests {
)]
);
}
+
+ /// Of the nodes `knows` names, those that didn't say they know a revision are named,
+ /// with what each said; a party whose nodes all did, not.
+ #[test]
+ fn nodes_not_known_to_know_a_revision_are_named() {
+ let one = Revision(1);
+ let id = one.id().to_string();
+ let counts = serde_json::json!({"parties": {
+ "lass": {"knows": {
+ "a": {"wire": [52, 52], "revision": {"number": 1, "id": id, "named": "one"}},
+ }},
+ "tv": {"knows": {
+ "b": {"wire": [52, 52], "revision": {"number": 0, "named": "the base"}},
+ "c": null,
+ "d": {"wire": [52, 52], "revision": {"number": null, "named": "x, unknown"}},
+ "e": {"wire": [52, 52], "revision": null},
+ "f": {"wire": [52, 53], "revision": {"number": 2, "named": "two"}},
+ }},
+ }});
+ assert_eq!(
+ unready(&counts, one),
+ [
+ "tv: b knows the base at newest; c said nothing: a build from before SPEAKS, \
+ or not heard from; d knows x, unknown; e knows no revision: built without \
+ the federation"
+ ]
+ );
+ assert!(unready(&counts, Revision(0)).len() == 1);
+ assert!(unready(&serde_json::json!({}), one).is_empty());
+ }
}
diff --git a/src/federation/party.rs b/src/federation/party.rs
index a12dc7f..2c8dc4b 100644
--- a/src/federation/party.rs
+++ b/src/federation/party.rs
@@ -314,7 +314,15 @@ pub(crate) async fn main() -> Result<()> {
moves to it, once every party's nodes run a continuum that knows it",
current.revision
);
+ warn_unready(&node.counts().await?, later);
}
+ } else if let Some(revision) = next["revision"]
+ .as_str()
+ .and_then(Id::from_hex)
+ .and_then(|id| Revision::of(&id))
+ && revision != current.revision
+ {
+ warn_unready(&node.counts().await?, revision);
}
println!("{}", serde_json::to_string_pretty(&next)?);
return Ok(());
@@ -755,18 +763,21 @@ async fn vote(
})) if hash == proposal => version,
_ => bail!("the node's {proposing} isn't the proposal it names"),
};
- // A later revision: the node knows it, or it stops at the version once it passes.
+ // A later revision: the node knows it, or it stops at the version once it passes; and
+ // so does every other node not known to know it.
if yes
&& let Ok(version) = Version::read(&proposed)
&& version.revision != current.revision
- && !node.revisions().await?.contains(&version.revision.id())
{
- bail!(
- "{proposal} moves to {}, which the node doesn't know: once it passed, a node that \
- doesn't stops at it, and votes nothing till upgraded; upgrade this node and every \
- other of the party to a continuum that knows it, then back it",
- version.revision
- );
+ if !node.revisions().await?.contains(&version.revision.id()) {
+ bail!(
+ "{proposal} moves to {}, which the node doesn't know: once it passed, a node \
+ that doesn't stops at it, and votes nothing till upgraded; upgrade this node \
+ and every other of the party to a continuum that knows it, then back it",
+ version.revision
+ );
+ }
+ warn_unready(&counts, version.revision);
}
// The votes on its party's earlier proposals it backed, for the nays among them.
let mut nays = Vec::new();
@@ -816,6 +827,21 @@ async fn vote(
node.put(&cast).await
}
+/// Warns of the nodes not known to know `revision`, by `counts` (`client::unready`).
+fn warn_unready(counts: &JsonValue, revision: Revision) {
+ let behind = client::unready(counts, revision);
+ if behind.is_empty() {
+ return;
+ }
+ eprintln!(
+ "Warning: these nodes aren't known to know {revision}, as far as the node heard from \
+ them; once a version of it passes, each stops at it till upgraded:"
+ );
+ for line in behind {
+ eprintln!(" {line}");
+ }
+}
+
/// Signs again what a move refused: of the latest event it signed at each place, its key
/// set and its node list, those the node refuses as not above its checkpoint's heads, or
/// as built on one it refuses.
diff --git a/src/swarm/speaking.rs b/src/swarm/speaking.rs
index dcbfac4..e1a8d2c 100644
--- a/src/swarm/speaking.rs
+++ b/src/swarm/speaking.rs
@@ -43,4 +43,30 @@ impl Speaking {
}
heard.insert(peer.to_owned(), speaks);
}
+
+ /// What `peer` said last, if it said anything.
+ pub(crate) fn of(&self, peer: &str) -> Option<Speaks> {
+ self.0.lock_or_recover().get(peer).copied()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn a_peer_is_heard_as_it_spoke_last() {
+ let speaking = Speaking::default();
+ assert_eq!(speaking.of("a"), None);
+ speaking.heard("a", own());
+ let older = Speaks {
+ oldest: 1,
+ newest: 1,
+ revision: None,
+ };
+ speaking.heard("b", older);
+ speaking.heard("a", older);
+ assert_eq!(speaking.of("a"), Some(older));
+ assert_eq!(own().newest, wire::VERSION);
+ }
}