path: root/docs/peers.md

# Peers: handshake, keepalive, dormancy and topology

The timings named below (`dormant_after` and the like) derive from `--keepalive`
([Timings](#timings), at the end).

The phases a node goes through (starting, joining, ready, rejoining, stopping) are in
[`lifecycle.md`](lifecycle.md); this page is the protocol between peers.

## Node IDs and addresses

A node is its node ID (`continuum:<hex>`, from its node key); a transport address only
says where it is (`mycelium:<public key>`). The identity layer (`transport::identity`)
between the node and its transports keeps the two apart, so everything below speaks of
node IDs. A node may have several transports, and reaches a peer on one both have
(module `transport::paths`). With several, it measures them: every 30 s (or every
keepalive interval, if that is longer) each gets a PING (the keepalive's, or one of its own), answered by a PONG over the same path, which
gives its round-trip time. A peer's only path is measured the same way once messages
go over it (a busy pair sends no keepalives). The
configured order decides until both are measured; then the node changes to another
path when it is a quarter faster (and at least 5 ms), so it doesn't flap. A send that
fails goes to the next path at once, and a path whose sends fail, or whose PINGs go
unanswered three times in a row, is down: used only when no other works. A path that
takes messages and drops them shows sooner: when what went to the peer goes unanswered
(the PING that asks again, below, or an overdue sync answer), its other paths come
first until that one answers a PING again, and all of them are pinged at once. A node on
several transports bridges nodes that share none ([`transports.md`](transports.md), Several transports):
the topology links it on each of its transports (below).

```mermaid
sequenceDiagram
    participant N as New node
    participant S as Seed
    N->>S: WHO(N's node key, signature over N's and S's address)
    S-->>N: ME(S's node key, signature over S's and N's address)
    Note over N,S: each knows which node is at the other's address
```

A WHO asks who is at an address and says who is asking; a ME answers. Each carries the
sender's public node key and its signature over the sender's address and the
recipient's. The transport authenticates the sender's address, so a proof holds for
that pair of addresses only. Once a node proved itself at an address, messages from
there are that node's; messages from an address no node proved itself at are dropped,
and the address is asked WHO (at most every 5 s), so after a restart the first message
from a peer brings its proof back.

A seed is an address (or something the transport resolves to one: a Mycelium public
key, IPv6 address or host name); it is asked WHO, and its ME names the node. Where other
nodes are comes from HELLO_REPLYs and the member list: hints, which are asked WHO before
anything is sent there; a send to a node that hasn't proven itself anywhere yet is
refused, and retried like any refused send. A node proven on some transports is also
asked WHO at its hinted addresses on the others, at most once a minute each: its ME
there adds a path to it (a second transport, Tor say), which is then measured and failed
over to. No seed is needed for that: the member list carries every member's addresses.

## Handshake and join

```mermaid
sequenceDiagram
    autonumber
    participant N as New node N
    participant S as Node S (reached via a seed)
    participant P as Peer P (known to S)

    N->>S: HELLO(swarm ID if known, epoch, key proof)
    Note over S: claimed swarm ID matches ours (or none claimed)<br/>N is member, Established, sync reset
    S->>N: HELLO_REPLY(swarm ID, active established neighbours [P], epoch, key proof)
    Note over N: swarm ID matches or is adopted<br/>S is member, Established, sync reset
    N->>P: HELLO(swarm ID, epoch, key proof) - discovered via S
    P->>N: HELLO_REPLY(...)
    par sync with one ready peer on the same epoch, the catch-up source (here S)
        S-->>N: sync messages
        N-->>S: sync messages
    end
    Note over N: all of the source's heads applied: initial sync done<br/>then sync with every neighbour on the same epoch
    alt S is on a newer epoch than N
        Note over N: no sync (epochs differ)<br/>remake worker starts a rejoin
        N->>S: BLOB request (snapshot change)
        S-->>N: manifest, then chunks (also from other peers on the epoch)
        Note over N: snapshot applied, then sync as above
    end
```

One round trip makes both sides members of each other: the HELLO proves the sender's
key, the reply the swarm ID and the responder's key. Sync messages don't repeat the
swarm ID: the handshake bound the peer to it. A new handshake resets the sync
session on both sides; HELLOs carry the handshake's number, so a resent HELLO (its
reply was lost) or a late copy is answered without resetting again. A node that doesn't know its swarm
yet (it is joining itself) doesn't answer a HELLO; the sender retries.

The reply lists at most 64 of the responder's active neighbours, each with the addresses
it reaches them at (at most 8): if it has more, those the recipient would most likely
pick itself (lowest pair score with it). The recipient takes at most 64 from a reply,
only valid identities (node IDs), and, once it knows the member list, only
its own neighbours; their addresses are hints for its transport, and each gets a
HELLO.

Every HELLO carries a random id of the sender's current run, and the HELLO_REPLY echoes
it. A reply with another id answers a HELLO from an earlier run (delivered late, or
queued in Mycelium while the node was down) and is ignored.

**Stale messages.** Mycelium may deliver a message up to 5 minutes late, and an external
Mycelium queues messages while our node is down (a restart drains that queue for up to
2 s, but only to cut noise). The protocol therefore must not depend on where in a
peer's message stream a node starts listening: unanswered HELLOs are retried, a HELLO
resets the session on both sides, sync messages are idempotent and checked against the
epoch, and lost or stale sync state is repaired by the reply timeout, the `need` rule and
the heal. Keepalives only inform.

Every message carries the sender's epoch, and with it how many times the sender changed
epoch in its current run. Sync messages, which go often and only between peers on one
epoch, carry a tag of it instead: its generation and the first 8 bytes of its ID. The
receiver takes it for its own epoch, or the sender's as it last knew it, whichever it
names; one that names neither is dropped. A node records a peer's epoch only from a message whose count
is at least the newest it has seen from that peer, so a message delayed from before
the peer switched (or rejoined, or gave up a rejoin) can't flip its recorded epoch
back. The count starts over when the peer restarts, which its HELLO or HELLO_REPLY
shows by a new run ID.

With a swarm key (`--swarm-key-file`), every HELLO and HELLO_REPLY carries
`HMAC-SHA256(key, label, swarm ID, from, to)`. A peer is admitted by its first valid one;
until then everything else it sends is ignored, and a handshake without a valid proof
gets no answer at all.

Retries: an unanswered HELLO is resent after 1 s, doubling up to 30 s, but at once
(at most every second) when anything else comes from the peer meanwhile: it is up and
reaches us, so the HELLO or its reply was lost, and waiting out the backoff (or a
probe's) would only delay the handshake. To an
unreachable peer (dormant, or three unanswered HELLOs) it is a probe: after
`dormant_probe`, doubling up to 6 × `dormant_probe` (or `--dormant-probe-max`, which is
every wait if shorter); members get probes even when they
aren't neighbours: those in the member list, as well as those whose HELLO_REPLY said so.
After a restart none has answered yet, and members that restart together may each fail
to reach the others at first (not up yet): without probes, none would try again. Nor
should they wait for one: a peer never heard from gets HELLOs at the retry backoff (up
to 30 s apart) for its first 5 minutes known, unreachable or not, and probes only after. A member a HELLO_REPLY lists among the sender's active neighbours is
probed at once: the sender reaches it, so after a partition the first pair to find each
other again brings the others' probes forward, and the sides reconnect within seconds
rather than pair by pair. So are all unreachable members once this node becomes
reachable where it wasn't (libp2p: a relay grants it a slot, or AutoNAT finds it
public): two nodes behind NATs that restart together can't reach each other until they
hold slots, and the second to get one reaches the first at once instead of at its next
probe. Sync messages from a peer that isn't a member yet are
dropped (the handshake resets the session anyway) and trigger our HELLO.

After a sync reset, both sides may open the sync; crossed openings cost nothing between
peers that share history. Only a ready node never opens towards a peer that isn't ready
(HELLO, HELLO_REPLY, sync messages and keepalives all say whether the sender is): that
peer's answer would carry empty heads, and the whole document would go out twice. The
peer that isn't ready opens itself.

While a node catches up (joining, or rejoining a newer epoch, until its initial sync is
done), it syncs with one peer only, the first it tries: otherwise every peer would send it
the whole document (or everything it lacks) at once. Messages from the others are applied
but not answered until the initial sync is done; then they only exchange heads. The
source counts as alive while any message from it arrives; after `catch_up_probe` of
silence (keepalive / 30, between 1 and 5 s) it gets a STATUS. After three unanswered
probes, or when it says it isn't ready, or twice that it wouldn't sync with us, the next
peer takes over. Being alive isn't progress, though: the next peer also takes over when
three of our sync messages in a row went unanswered and the source has sent no sync
message for the transport's delivery limit (five minutes for Mycelium: by then it has
delivered or dropped anything sent, however large); the source given up then isn't
chosen again for as long,
so a stalled source doesn't simply take the role back, and gets a fresh HELLO, whose
reply brings its current peer list (the first one may have listed nobody: a peer lists
only its neighbours, and a new member isn't one until its registration has spread). Every minute without a sync
message from the source is logged as a warning.

A sync message of 1 KiB or more goes deflated, if that makes it at least a tenth
smaller: Automerge sends changes as they are stored, and text or JSON in them shrinks to
about half. The receiver inflates it to no more than the largest sync message it
accepts.

A sync message larger than 16 chunks (16 × `--blob-chunk`, 1 MiB by default) isn't
sent: it would be all or nothing, sent again whole after a lost copy, and on a slow link
may not arrive within the transport's delivery limit (Mycelium's five minutes). The sender takes it back and offers a
*delta* instead: the changes in the history of its heads but not of the peer's (as far
as it knows them; of those, the ones it holds, and for any it lacks, the heads they
share). The peer fetches them as a blob, in chunks (each deflated where that makes it
at least a tenth smaller), from the sender and any other peer holding both sets of
heads (every such peer produces the same bytes), loads them,
and syncs on; the rest goes in ordinary messages. The offer is the sender's answer to
what the peer sent: the peer waits for no other while it downloads. Until the peer answers, the sender
sends it nothing; if the answer is overdue, it offers again (the same offer, without
generating the message again, while neither side's heads moved). This is how a join without a
recent remake, or a return after much was written, gets the bulk of its changes. A node
downloads one delta at a time: while a large one comes, it lacks those changes, so its
other neighbours offer them too, and they would come once per offer. Offers that come
meanwhile wait; once the delta is in, the node syncs with those that offered, which then
send only what is still missing. What a delta brought counts as known to its sender, as
what its sync messages bring does: none of it goes back, by message or as a delta.
An offer made before its sender heard what the node has now (one that came behind the
chunks of the delta before it, say) would bring part of that again: such an offer isn't
fetched; the node's next message tells the sender, which offers afresh if still needed.

Towards a peer on a slow link the limit is lower: a message is offered as a delta if it
would take more than 5 s at the throughput measured to that peer (`syncThroughput`,
[`measurements.md`](measurements.md)), or if it is at least half as large as one that didn't make it to
the peer in the last hour (its send failed, or it went unanswered; only those of more
than two chunks count), but never below one chunk. A message too large for the link
then goes chunk by chunk instead of failing and being sent again whole.

Every sync message says whether its sender is ready. Only a ready peer can be the source,
and only a message from one completes the initial sync: a peer that is itself still
catching up has empty heads, which any document trivially contains.

The two sides of a pair take turns sending keepalives; the one that leads the pair sends
the first. The leader is chosen by one bit of `SHA-256(label, smaller key, larger
key)`, so both sides agree without talking and each node leads about half of its pairs
([`swarm/pair.rs`](../src/swarm/pair.rs)).

## Liveness, dormancy and HELLO state (per peer)

```mermaid
stateDiagram-v2
    direction LR
    state Liveness {
        [*] --> Unknown
        Unknown --> Active: any message received
        Unknown --> Dormant: node ready, a neighbour, nothing received for dormant_after
        Active --> Dormant: node ready, a neighbour, nothing received for dormant_after
        Active --> Dormant: its farewell (a STATUS saying it is leaving)
        Dormant --> Active: any message received (after its farewell: from its next run)
    }
    state Handshake {
        [*] --> NotSent
        NotSent --> Sending: HELLO goes out
        Sending --> AwaitingReply: sent (or failed)
        AwaitingReply --> Sending: retry due
        AwaitingReply --> Established: HELLO_REPLY
        AwaitingReply --> NotSent: no longer a neighbour (unless probed while unreachable)
        Sending --> Established: HELLO_REPLY
        Established --> NotSent: heal, stall reset, peer reactivated,<br/>or it no longer proves the swarm key
    }
```

- **Unknown** is a peer nothing was heard from since this node started (it is known
  from the member list, or another peer named it). It is treated as active; it only
  shows that nobody answered yet.
- **Becoming dormant** resets sync with the peer, stops syncing and pinging it, and
  probes it with a HELLO (after `dormant_probe`, doubling). A dormant member stays in
  the electorate of remakes while its alive stamp is recent ([`remakes.md`](remakes.md)), so enough
  of them make a majority impossible; an operator can forget them (`DELETE /peers/{peer}`,
  a swarm-wide record that is cancelled if the peer turns out to be alive).
- **Any message** from a dormant peer makes it active again and forces a fresh
  handshake, whose reply resets sync on both sides.
- Dormancy is only judged while the node is ready, so joining nodes keep fast retries.
  A peer's farewell (it is shutting down: a STATUS with `leaving`) makes it dormant at
  once, whatever our phase; it isn't counted as an incident (`dormancies`). What that
  run still sends while it stops (sync messages, PINGs, a LINKS_REPLY) doesn't make it
  active again: only a HELLO or HELLO_REPLY from its next run does.
- **Strangers** (peers that never became members: listed to us but never answering, or
  failing the handshake) are forgotten locally once silent for a day, so peers named
  to a node can't grow its list without bound.

## Member registry and neighbours

`_continuum/members/<node ID>` lists every node that has joined, replicated with the
document and carried into new epochs. An entry says where its node can be reached
(`addresses`, kept up to date by the node), so members find each other without having
met. It is a pure registry, not access control (that
is the swarm key). A node registers itself as soon as it is ready and again whenever its
entry is missing while it runs (checked every 5 s). Forgetting a peer (`DELETE /peers/{peer}`)
removes its entry along with writing the forget record; a forgotten node that is alive
cancels the record and registers again. Each node acts on a record as soon as it has
it (and looks at the records again every 5 s), once: a peer it hears of again before
the cancel arrives is kept.

From the member list every node computes the graph ([`swarm/topology/`](../src/swarm/topology/)), leaving out
members it can't reach (three unanswered HELLOs, or dormant, until they answer a probe;
or with no address on any of its transports), and those whose alive stamp is older than
the electorate window (`electorate-window`) that it hasn't heard from since it started
(a member coming back sends its HELLO, and counts again from then):
a ring by key hash, each member linked to the nearest reachable member
in both directions, plus, per member, the `k − 2` reachable others with the lowest pair
score; a link exists if either end chose it. With several transports, each has such a
graph among the members on it (by the addresses in their entries), and a node's
neighbours are its neighbours on each transport it is on: every transport's graph is
connected, and a member on two transports is in both, so as long as some members are
on several, the swarm is connected, and those have about `k` neighbours per
transport. With 100 members of which 10 are online,
the 10 form a full mesh; nodes that briefly disagree on who is reachable only add links. `k` is the swarm's: its genesis node records its
`--neighbours` (default 12) in the document (`_continuum/neighbours`), and every node
builds the graph with that value, since every node must build the same one (the
writers' trees are built on it); a protocol writer may change it later, and every node
then recomputes its neighbours and the trees. A swarm from before it was recorded has
none, and each node uses its own `--neighbours`, which must then be the same
everywhere. So swarms of up to `k` + 1 nodes (13 by default) are a full mesh. A node *initiates* only towards its neighbours (HELLO, keepalive, dormancy
checks, sync openings, pushing its changes, epoch announcements, the heal, the peer list
in HELLO_REPLY) and *answers* anyone. Changes travel over several hops; heads in
keepalives tell a node when a neighbour has something it lacks. Until a node is ready
and knows the member list, everyone counts as a neighbour, so joining works as before.
A peer that proved itself a member of the swarm (its handshake) but isn't in the member
list yet is a neighbour too, until it is: otherwise a node that joined before the
genesis node registered itself, and registered itself in a document without the genesis
node, would never sync with it, and neither registration would pass.
So is a member that lists no address on the node's transports but reaches it anyway, as
long as it says it syncs with the node (STATUS): it only connects out (a libp2p node
without a listen address), so it is in no graph but its own, and the nodes it picked
would otherwise only answer it, never send it their changes.
The neighbours are recomputed as soon as the member list changes or a peer becomes
unreachable or reachable again, and every 5 s for what has no event of its own
(unreachability expiring); new ones get a HELLO and a sync opening, former ones a session
reset.

Each node judges its own stability ([`swarm/stability.rs`](../src/swarm/stability.rs)): the fraction of the last 24 h
it had at least one live peer. It writes only a coarse tier into its registry entry
(`tier`: `stable` after a day at ≥ 95%, back to `transient` below 80%), and only when it
changes. After a restart, a stable node stays stable if a peer's HELLO_REPLY says it was
away for less than an hour; otherwise it starts over as transient. In the neighbour graph,
half of each member's scored slots prefer stable members, so long-running nodes form the
backbone (with 20 stable of 250 members and k = 12: about 65 links per stable member).

Changes travel through a partial mesh by announcement and pull ([`swarm/relay.rs`](../src/swarm/relay.rs)): a node
pushes its own writes to its neighbours; after applying changes from one neighbour it
sends its other neighbours a STATUS with its new heads (marked as an announcement: no
keepalive, and answered only by a neighbour that lacks some of those heads, so one that
has the changes already sends nothing back, even if its heads differ, as they do while
writes go on and newer changes are still on their way to it; nor is a neighbour told whose latest heads, as its
own announcement or STATUS brought them, are this node's: announcements wait 200 to
400 ms, at random, so of two neighbours with the same news one is usually first, and
the other skips it; nor one whose latest heads, as it said them in a STATUS or a sync
message, have all of this node's in their history: it is as far or further), and a neighbour that lacks those heads pulls them (it sets them as the peer's heads in its sync state, so Automerge asks
for what's missing), if it still lacks them after a round trip to the announcer (at
most 1 s): while writes go on, the newest are usually on their way to it along a
writer's tree, and would otherwise come twice. It pulls from one neighbour at a time
(until the answer is overdue, by the sync reply timeout, or 15 s); neighbours that
announce heads it lacks meanwhile wait their turn and are pulled from as soon as the
current pull is answered or overdue, if they still have something it lacks. The regular
STATUS (every third keepalive) triggers the same pull, so gaps close within a few
keepalive intervals. Changes that pass on along a writer's tree (below) go at once,
without announcement or pull; the announcements catch what the trees miss. A node
that just caught up announces its heads once. In a full mesh too: the member list makes
every member the writer's neighbour, but the writer may reach fewer of them (one behind
NAT may reach a single one, one on libp2p alone none on Tor alone), and no node can tell
which from where it is. A neighbour that has the changes already only answers.

Every node tells its neighbours whom it reaches ([`swarm/reach.rs`](../src/swarm/reach.rs)): the members it
has an active session with (handshake done, neither dormant nor unreachable) that
answer, each by an 8-byte key (the hash its pair scores come from) and its round trip
to it as a coarse bucket (doubling bounds from 25 ms to 800 ms; one that moves only
once 20% past a bound, so a round trip near one doesn't flip; one never measured in the
middle), in its keepalive STATUSes and their replies, and within 5 s when either changed; not in announcements of
heads, which go often. A member that leaves a keepalive or sync message unanswered,
with nothing else heard from it, is asked again (a PING) after the first sync reply
timeout, and after twice that no longer counts as reached: its neighbours are told at
once, and the members whose parent it was in a writer's tree take another, within
seconds rather than after `dormant_after`. Any message from it, and it counts again. So every node knows its 2-hop neighbourhood as measured, not as the member list
says, for as long as each neighbour's session lasts (one that goes dormant or starts
anew says it again), and nothing goes into the document. Not for a set time: while a
pair exchanges changes it sends no keepalives, and so no STATUS, however long the
writes go on. `GET /peers` shows it
(`localReaches`, and `reaches` per peer).

A writer's changes pass on along its tree ([`swarm/tree.rs`](../src/swarm/tree.rs)), the same on every node:
built on the graph the member list gives (as if every member reached every other) and
on whom each member says it reaches. Members are ordered by their distance from the
writer over that graph, then whether they reach a member a hop closer, then by key; a
member's parent is the neighbour it reaches that comes before it, the closest to the
writer first (a link at least two round-trip buckets slower than its fastest to one
counting a hop more: a slow circuit to the writer is passed over for a fast link to a
member as close), then one that reaches closer, then the one it said it has the shortest
round trip to, then by pair score. Coming before rules out
cycles; taking only one it reaches routes around links that don't work: where a writer
behind NAT reaches a single member, that one comes first among those as far from it,
and the others take it as their parent. Whose a change is, its author says (every
node writes as itself, its node ID): whoever applies changes sends them at once to its
children in each of their writers' trees, however they came (along a tree, with an
answer to one of its own messages, pulled). A sync message along a tree also says
whose; one that had those changes already passes them on too (they may have come
another way first, when it wasn't yet anyone's turn to pass them on), so a child that
lacks them still gets them. A sync message to a peer that had nothing the sender lacks,
with no answer awaited, needs none: the sender takes it as delivered (the peer's heads
are its own now) and says so in the envelope, and the peer, level with it, says
nothing; one with changes the sender lacks answers with them, as before. That leaves
out about half the sync messages, Automerge's answers that only say "I have your
heads": along trees, to pulls, to what is sent again. Should the message be lost, the
peer still lacks it: one that pulled asks again after its reply timeout; a child in a
tree is told by its other neighbours in their slow rounds (below), and pulls (one that
reaches no other member, by its parent's), or can't apply the next message and asks for
what it lacks, and gets just that, and what that builds on, again (not what was sent
after it: that may only be late).

A node's neighbours that get a writer's changes from another parent in its tree, and
the writer, count as having them: what it passes on to them later along other trees
doesn't carry them again. Without that, with writers in turn, each change coming along
another tree, the node's view of a neighbour stays as of its last message to it, and
the next carries all it got since, nearly all of it brought by the neighbour's other
parents (with 50 nodes and writers in turn, about ten changes per message, one of them
new). Should one lack them after all, it asks, as above. But what the node passes on
to a child carries those of them the passed-on changes build on, if counted so within
the last 2 s: with writers in turn each change builds on the last writer's, which may
still be on its way along another tree, and the child would have to ask for it, a round
trip later.

As nobody waits for these answers, messages overtake each other on slow links. Three
things keep that from sending changes twice (`document::sync`):

* A message names what its sender last shared with the receiver by heads, and for one
  taken as delivered, those are the heads its predecessor brought. Where the
  predecessor is still on its way, the receiver lacks them; Automerge would take that
  for lost data and answer with a reset, after which the sender sends everything
  again. So heads the receiver lacks give way to those it shares with the sender.
* The changes a message brought count as already sent to its sender, which has them:
  an answer never carries them back.
* A message that turns out older than the last from its sender, once all of it is in,
  leaves the receiver level with the newer one: it says nothing.

The announcement round of a node that passed changes on goes only to the neighbours the
tree leaves out (no member they reach comes before them, or the member list's graph
doesn't have them); all others are told in a slow round 2 s later, which catches what a tree missed
(a parent that went away, nodes whose views of the tree differed, the last of a run of
messages along a tree lost) and covers everything that came meanwhile: while changes
keep coming, each neighbour gets an announcement about every 2 s, not with every
change. As a neighbour answers an announcement only when it lacks something, an
answered slow round means a tree missed; while none is answered, the trees work, and
each slow round waits twice as long as the one before, up to 16 s, and 2 s again after
the first answer.
A member without a parent it reaches, or whose parent missed the change, gets it by
announcement and pull as before, then passes it on to its own children. With every link
working, each member gets each change once (`bench_writer_trees`: 1999 messages per
change at 2000 members, against about 25,000 by announcements alone, at most 5 hops);
with 5% of the links down, more than 90% get it along the tree, the rest after one or
two announcement rounds.

## Keepalive and self-healing

```mermaid
sequenceDiagram
    participant A as Node A (its turn)
    participant B as Node B

    Note over A: B established and active, quiet for keepalive,<br/>no keepalive to B for keepalive, known for keepalive
    alt no STATUS between them for 3 × keepalive
    A->>B: STATUS(heads, epoch, ready, syncs with you)
    Note over B: mark A seen (reactivate if dormant)<br/>record A's epoch, reset sync if it changed<br/>re-handshake if A was dormant or unknown
    B->>A: STATUS_REPLY(heads, epoch, ready, syncs with you)
    Note over A: mark B seen, record B's epoch
    else otherwise
    A->>B: PING(nonce)
    Note over B: mark A seen (reactivate if dormant)
    B->>A: PONG(nonce, B's time)
    Note over A: mark B seen, round-trip sample
    end
    alt after a STATUS: same epoch, heads have differed for resync_after with no sync from it since<br/>nor both sides' heads moving on, no answer due to our own sync message, last heal at least resync_after ago
        A->>B: HELLO
        B->>A: HELLO_REPLY
        Note over A,B: sync reset on both sides, sync restarts
    end
    Note over A,B: after an epoch switch a node announces its epoch at once:<br/>ready peers already on it get its sync opening,<br/>all others a STATUS#59; a node that becomes ready<br/>sends every neighbour a STATUS
```

A quiet pair's keepalive is a PING, which the peer's paths layer answers at once with a
PONG over the same path (`transport::paths`): both sides learn the other is alive, and
the leader gets a round-trip time. Every third one is a STATUS instead, answered with a STATUS_REPLY: each
side's state, for the heal and against lost announcements. A node's state also goes out
when it changes: its heads (announcements, above), its epoch after a switch, and its
readiness when it becomes ready (or ready again after a rejoin) and at shutdown.

A STATUS lists the sender's heads (up to 32; more as a digest of them with 32 of
them, another 32 each time, so a recipient pulls the listed ones it lacks), its epoch,
whether it is ready, and whether it would sync with the recipient.

The two sides take turns: after a keepalive from the peer, the next one is ours (the
leader of the pair sends the first). On its turn a node sends once the pair has been
quiet for a keepalive interval; off its turn only after 1.5 × keepalive, in case the
other one doesn't, e.g. because a keepalive or its answer was lost (any message counts
as heard, so sync traffic suppresses keepalives on both sides). If two keepalives
cross anyway (the peer's arrives within half an interval after ours), the next turn is
the leader's, so they don't keep crossing. One keepalive and its answer per pair and interval, each side sending
every other one, so both measure the round trip, instead of keepalives from both sides
that often cross.

Inside sync, a lost message is resent after the reply timeout (10 s, 30 s with Tor
enabled, doubling up to 30 times that). If the peer asks for a change we believe it has (the message carrying it was
lost while later ones arrived), we forget which changes were sent, and the next message
carries everything it lacks. A peer that got a change without the ones it builds on asks
only for those, without saying what it has; Automerge would send just them, and the peer
ask again for theirs, one round trip per missing change. So we answer as if it had
nothing since what we share: everything since then goes at once (a blob, if large).

The heal comes last, under all of this: only where sync is idle on our side (nothing
awaited) while heads stay apart. A pair whose sync messages are lost keeps awaiting
answers, resending with each reply timeout, and doesn't heal: a re-handshake would be
lost the same way. What the heal is for is a session out of step that nothing above
notices; no scenario of the bench, lossy, slow or with members away, needs one.

## Timings

Derived from `--keepalive` (`Timing::from_keepalive`, [`state/timing.rs`](../src/state/timing.rs)):

| Name | Value | What for |
|---|---|---|
| `status_refresh` | 3 × keepalive | every third keepalive is a STATUS |
| `dormant_after` | 4 × keepalive | silent this long, a peer is dormant (three keepalives lost in a row are tolerated) |
| `resync_after` | 2 × keepalive, at least twice the first sync reply timeout (20 s; 60 s with Tor) | heads differing this long make a heal |
| `dormant_probe` | 10 × keepalive | the first probe of an unreachable peer, doubling up to 6 × (or `--dormant-probe-max`) |
| `catch_up_probe` | keepalive / 30, from 1 to 5 s | how soon a silent catch-up source is probed |
| sync reply timeout | 10 s (30 s with Tor enabled: the longest of the transports'), doubling up to 30 × | resending a sync message nothing answered |
| `unreached_after` | 2 × the first sync reply timeout | a keepalive or sync message unanswered this long (asked again halfway), a peer no longer counts as reached |

The peer worker runs when the earliest of these is due for some peer (and when something may have made one due sooner: a HELLO sent, a sync message sent, a keepalive received, a new peer or new neighbours), at least every 5 s, and tells the neighbours whom it reaches at once when a member it reached stops answering; the remake worker every second.