announcements answered when lacking

A neighbour now answers an announcement only if it lacks some of the
announced heads.  Heads that merely differ are no longer enough, since
they do while writes go on.  The check reads the document, so it runs
off the message consumer, as the pull check already does.

7 files changed, 40 insertions, 15 deletions

diff --git a/docs/architecture.md b/docs/architecture.md
index 0ef4f37..2ad7638 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -235,7 +235,7 @@ tree reaches them, else in the slow round 2 s later.
 replies, admit (swarm key), mark the peer seen, record its epoch (ignoring late ones),
 then by type: HELLO answered, HELLO_REPLY establishes membership and discovers peers,
 sync queued on its lane if on our epoch (held back from everyone but the source while catching
-up), STATUS answered with a STATUS_REPLY (an announcement only if the heads differ) and compared for divergence, blobs served or collected,
+up), STATUS answered with a STATUS_REPLY (an announcement only if the receiver lacks some of its heads) and compared for divergence, blobs served or collected,
 LINKS answered with the node's links if the asker is a member, and again on each change while it subscribes, or with other members' links it asks for the asker, which can't reach them (`swarm::links`, for the dashboard);
 PING and PONG (the paths layer answered the PING already) only mark the peer seen, and
 after a PONG the peer's round-trip time is its path's again.
diff --git a/docs/peers.md b/docs/peers.md
index c2f1ddb..8b58df0 100644
--- a/docs/peers.md
+++ b/docs/peers.md
@@ -281,8 +281,9 @@ backbone (with 20 stable of 250 members and k = 12: about 65 links per stable me
 Changes travel through a partial mesh by announcement and pull ([`swarm/relay.rs`](../src/swarm/relay.rs)): a node
 pushes its own writes to its neighbours; after applying changes from one neighbour it
 sends its other neighbours a STATUS with its new heads (marked as an announcement: no
-keepalive, and answered only by a neighbour whose heads differ, so one that has the
-changes already sends nothing back; nor is a neighbour told whose latest heads, as its
+keepalive, and answered only by a neighbour that lacks some of those heads, so one that
+has the changes already sends nothing back, even if its heads differ, as they do while
+writes go on and newer changes are still on their way to it; nor is a neighbour told whose latest heads, as its
 own announcement or STATUS brought them, are this node's: announcements wait 200 to
 400 ms, at random, so of two neighbours with the same news one is usually first, and
 the other skips it), and a neighbour that lacks those heads pulls them (it sets them as the peer's heads in its sync state, so Automerge asks
diff --git a/src/swarm/peer/inbound.rs b/src/swarm/peer/inbound.rs
index 1454c3e..28066cb 100644
--- a/src/swarm/peer/inbound.rs
+++ b/src/swarm/peer/inbound.rs
@@ -143,13 +143,31 @@ pub(crate) async fn handle_message(
             crate::swarm::blob::handle(state, &peer, blob).await;
         }
         transport::ParsedMessage::Status(remote) => {
-            // An announcement is no keepalive; it is answered only if our heads differ,
-            // so the announcer learns what we have that it lacks.
+            // An announcement is no keepalive; it is answered only if we lack some of the
+            // heads it announced (`relay::lacks`), so the announcer learns what we have
+            // that it lacks while we pull. Having all of them, we are as far or further,
+            // and what we have further we announced ourselves; differing heads alone are
+            // no reason, as changes on their way to us make them differ while writes go on.
             if !remote.announcing {
                 status_received(state, &peer).await;
             }
             let local = on_status(state, &peer, &remote, reactivated).await;
-            if remote.announcing && local.heads == remote.heads {
+            if remote.announcing {
+                if local.heads != remote.heads {
+                    // Off the consumer: telling which heads we lack takes the document's
+                    // lock.
+                    let (state, peer) = (state.clone(), peer.clone());
+                    tokio::spawn(async move {
+                        if crate::swarm::relay::lacks(&state, &remote.heads).await
+                            && let Err(error) = state
+                                .transport
+                                .send_message(&peer, &ParsedMessage::StatusReply(local))
+                                .await
+                        {
+                            debug!(peer = %peer, error = %format!("{error:#}"), "STATUS_REPLY failed");
+                        }
+                    });
+                }
                 return Ok(());
             }
             if let Err(error) = state
diff --git a/src/swarm/peer/keepalive.rs b/src/swarm/peer/keepalive.rs
index 8362d73..f4e961f 100644
--- a/src/swarm/peer/keepalive.rs
+++ b/src/swarm/peer/keepalive.rs
@@ -33,8 +33,8 @@ pub(crate) async fn send_status(state: &AppState, peer: &str) -> Result<()> {
 }
 
 /// Sends `peer` a STATUS announcing our new heads (`swarm::relay`): no keepalive, so it
-/// leaves the keepalives' schedule as it is, and it is answered only if the peer's heads
-/// differ (`Status::announcing`). Whom we reach only `with_reaches` (`swarm::reach`):
+/// leaves the keepalives' schedule as it is, and it is answered only if the peer lacks
+/// some of our heads (`Status::announcing`). Whom we reach only `with_reaches` (`swarm::reach`):
 /// announcements of heads go often.
 pub(crate) async fn announce_heads(state: &AppState, peer: &str, with_reaches: bool) -> Result<()> {
     let local = local_status(state, peer).await;
diff --git a/src/swarm/reach.rs b/src/swarm/reach.rs
index 7bd0f73..5c89d04 100644
--- a/src/swarm/reach.rs
+++ b/src/swarm/reach.rs
@@ -72,8 +72,8 @@ pub(crate) fn published(state: &AppState) -> Vec<u64> {
 
 /// With the peer worker's periodic pass (every 5 s), and at once when a member we said
 /// we reach stops answering (`reached_stops_answering`): whom we reach now; if it changed,
-/// our neighbours are told at once (a STATUS each, unanswered if the heads are the
-/// same). A change that is undone within a pass isn't told.
+/// our neighbours are told at once (a STATUS each, unanswered unless they lack some of
+/// our heads). A change that is undone within a pass isn't told.
 pub(crate) async fn check(state: &AppState) {
     let now = measure(state).await;
     {
@@ -109,8 +109,8 @@ pub(crate) fn reached_stops_answering(state: &AppState, peer: &str) -> bool {
         .is_some_and(|keys| keys.contains(&topology::pair_key(peer)))
 }
 
-/// A STATUS to `peer` with whom we reach: as an announcement (unanswered if the heads
-/// are the same), but with `reaches`.
+/// A STATUS to `peer` with whom we reach: as an announcement (unanswered unless it lacks
+/// some of our heads), but with `reaches`.
 async fn tell(state: &AppState, peer: &str) -> anyhow::Result<()> {
     announce_heads(state, peer, true).await
 }
diff --git a/src/swarm/relay.rs b/src/swarm/relay.rs
index 53c1065..99c1381 100644
--- a/src/swarm/relay.rs
+++ b/src/swarm/relay.rs
@@ -5,7 +5,7 @@
 //! further by itself but along a writer's tree. So after applying changes from one
 //! neighbour, a node *announces*
 //! its new heads to its other neighbours with a STATUS (heads, about 100 bytes, not the
-//! changes; no keepalive, and answered only by a neighbour whose heads differ). A
+//! changes; no keepalive, and answered only by a neighbour that lacks some of them). A
 //! neighbour that sees heads it lacks *pulls*: it enters the
 //! announced heads as the peer's heads in its sync state, Automerge asks for what's
 //! missing (`need`), and the announcer answers with exactly those changes. Regular
@@ -218,6 +218,12 @@ pub(crate) struct Pulling {
     waiting: HashMap<String, transport::Status>,
 }
 
+/// Whether we lack any of `heads` (with a digest, any of the sample it lists).
+pub(crate) async fn lacks(state: &AppState, heads: &HeadsSummary) -> bool {
+    let replica = state.doc.read().await;
+    !crate::document::changes::missing(&replica.doc, heads.listed()).is_empty()
+}
+
 /// A keepalive from `peer`: if it has heads we lack, pull them (one peer at a time; a
 /// busy slot queues the peer). Returns whether a pull started.
 pub(crate) async fn pull_if_behind(
diff --git a/src/transport/mod.rs b/src/transport/mod.rs
index e27ce2d..17d6d4c 100644
--- a/src/transport/mod.rs
+++ b/src/transport/mod.rs
@@ -203,8 +203,8 @@ pub(crate) struct Status {
     /// recipient holds it dormant at once, rather than after `dormant_after` of silence.
     pub(crate) leaving: bool,
     /// The sender announces new heads (`swarm::relay`), rather than keeping alive: a
-    /// STATUS answered only if the recipient's heads differ. Most announcements reach a
-    /// neighbour that has the changes already, which has nothing to tell.
+    /// STATUS answered only if the recipient lacks some of the heads. Most announcements
+    /// reach a neighbour that has the changes already, which has nothing to tell.
     pub(crate) announcing: bool,
     /// Whom the sender reaches (`swarm::reach`): the pair keys (`topology::pair_key`)
     /// of the members it has an active session with that answer, sorted. In keepalive STATUSes and