add documentation

1 files changed, 299 insertions, 0 deletions

diff --git a/docs/measurements.md b/docs/measurements.md
new file mode 100644
index 0000000..2adc0f5
--- /dev/null
+++ b/docs/measurements.md
@@ -0,0 +1,299 @@
+# Measurements
+
+What a node measures about itself and its peers, where to read it, and what the
+numbers mean. Everything here comes from traffic and work the node has anyway; it
+sends no probes of its own and nothing is measured by anyone else, but for the
+dashboard's map (LINKS, only while someone looks at it).
+
+Four places to read:
+
+* `GET /admin/stats`: the node's figures as JSON, or as Prometheus text when asked
+  for it (`Accept: text/plain` or `application/openmetrics-text`). Prometheus names
+  are `continuum_…`; per-peer series carry a `peer` label (the node ID) and can be
+  joined with `continuum_peer_info` for the alias.
+* `GET /peers`: one entry per known peer, keyed by its node ID, with the per-peer
+  figures.
+* The log: incidents as they happen (a peer going dormant, a heal, a stalled catch-up),
+  at warn or info; `PUT /admin/log` changes the filter at runtime.
+* `GET /admin/dashboard` (builds with the `dashboard` cargo feature): a page with the
+  two above and a map of the swarm, live ([`dashboard.md`](dashboard.md)).
+
+All counters count since the process started; a restart resets them. Durations are
+whole seconds in JSON unless named `…Ms`; Prometheus has seconds throughout.
+
+## About the node
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `uptimeSecs` | `uptime_seconds` | Since the process started. |
+| `ready` | `ready` | The node holds the swarm's document and serves writes ([`lifecycle.md`](lifecycle.md)). |
+| `epoch` | `epoch_generation` | The local epoch; the generation counts remakes since genesis. |
+| `staleHelloReplies` | `stale_hello_replies_total` | HELLO_REPLYs to a HELLO from an earlier run of this node, ignored. A few after every restart are normal (a transport queued them, Mycelium's above all); a steady stream means a peer keeps answering an old HELLO. |
+
+## The document
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `document.heads` | `document_heads` | Heads of the local document now. 1 means every change has been merged into one line of history; more means concurrent writes not yet followed by a merging write. |
+| `document.maxHeads` | `document_max_heads` | The most heads seen right after applying a sync message. A high value shows how concurrent the swarm's writing gets; it also bounds keepalive size (above 32 heads, keepalives carry a digest and 32 of them). |
+| `document.savedBytes` | `document_saved_bytes` | The document as Automerge saves it (compressed): roughly what a joining node downloads and what a document file rewrite writes. Measured in the background, only after a change, at least a minute apart and at most 1% of the time (a document that takes 2 s to save is measured every 200 s at most); an export (`GET /admin/document`) counts as a measurement. `savedBytesSecsAgo` and `savedBytesTookMs` say how fresh and how costly. |
+| `document.lockWait.read`, `.write`: `count`, `sumUs`, `maxUs`, `buckets[]` (`leUs`) | `document_lock_wait_seconds{mode}` (histogram), `document_lock_wait_max_seconds{mode}` | How long takers of the document's lock waited for it, since start, by mode. Readers (the API's reads, topology, stats, generating sync messages, serving changes and deltas, saves) wait only for a writer; writers (applying sync messages and fetched changes, writes, epoch switches) wait for everyone. Buckets from 10 µs to 10 s, the last unbounded; not cumulative in JSON. |
+
+**Interpretation.** `savedBytes` growing without bound is the reason to compact
+([`remakes.md`](remakes.md)); `savedBytesTookMs` is a direct measure of what a switch, a rejoin and a
+document file rewrite cost in CPU.
+
+The node has one document, and nearly all work on it takes its lock, so `lockWait` says
+whether more cores would help: waits of a few microseconds mean the lock is free when
+asked for; write waits in the milliseconds (`sumUs` / `count`), growing with the number
+of neighbours, mean applying one peer's sync messages queues behind the others'. A
+high `maxUs` alone is a single long hold (a large sync message, a remake), not
+contention.
+
+## Catching up
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `catchUp.sources` | `catch_up_sources_total` | Catch-up sources chosen. 1 is normal; more means the first source was given up (quiet, not ready, refused, or stalled) and another took over. |
+| `catchUp.doneAfterSecs` | `catch_up_duration_seconds` | From start until the initial sync was done; absent until then, and on a node that was ready at once (genesis, document file, recovery). |
+| `catchUp.syncBytes` | `catch_up_sync_bytes` | Sync payload received until then, from every peer: what joining cost. |
+
+Compare `syncBytes` with `savedBytes`: much more received than the document is large
+means the node got the document from several peers before the source rule took hold,
+or caught up over many small messages.
+
+## Remakes
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `remakes.undecided` | `remakes_undecided` | Remakes of the local epoch not decided yet, expired ones included (`GET /admin/remakes`). |
+| `remakes.committed` | `remakes_committed` | Committed remakes of the local epoch: 1 for the moment between a commit and the switch. More than one is a double commit, on which no node switches by itself: alert on it. |
+| `remakes.rejoining` | `rejoining` | The node syncs a newer epoch from scratch ([`lifecycle.md`](lifecycle.md)). |
+| `remakes.catchUpAttempts` | `catch_up_attempts` | Attempts at catching up with a newer epoch; 0 when not catching up. A growing count means the node can't get the epoch its peers advertise. |
+| `remakes.quarantined`, `remakes.oldestQuarantineSecs` | `quarantined`, `oldest_quarantine_seconds` | Change sets set aside on a rejoin (`GET /admin/quarantine`). They live in memory only, so a restart loses them: alerting is how to learn of them. |
+| `remakes.archivedEpochs` | `archived_epochs` | Old epochs' documents kept in memory (`archiveRetention`). |
+
+## Sync and relay
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `sync.lastFromNeighbourSecs` | `last_sync_from_neighbour_seconds` | Since a sync message from any neighbour was applied. Growing without bound while others write: the node is cut off from the swarm's changes. |
+| `sync.divergedPeers` | `diverged_peers` | Peers whose heads differ from ours with no sync message in between. |
+| `sync.largestReceivedBytes` | `largest_sync_received_bytes` | The largest sync message received since start. Larger transfers come as deltas (blobs of 16 chunks or more, [`peers.md`](peers.md)), so this stays below 16 × `--blob-chunk` from nodes of this version. |
+| `relay.announceRounds` | `announce_rounds_total` | Announcement rounds: our new heads to the neighbours after applying their changes (partial mesh only). |
+| `relay.pulls` | `pulls_total` | Pulls of changes we lacked from a neighbour that announced them. |
+| `relay.pullsQueued` | `pulls_queued_total` | Announcements of heads we lacked that waited for the pull in progress. Many compared with `pulls`: changes arrive from several sides at once. |
+| `relay.blobRequestsDropped` | `blob_requests_dropped_total` | Blob requests not answered because too many were being answered; the requester asks again or asks another peer. Also logged as a warning, at most once a minute with the count. |
+
+## Topology
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `topology.k` | `neighbours_target` | `--neighbours`. |
+| `topology.view.members` | `members` | Registered members, this node included. |
+| `topology.view.reachable` | `reachable_members` | Members other than this node not judged unreachable (three unanswered HELLOs, or dormant). |
+| `topology.view.stable` | `stable_members` | Members whose registry entry says `stable`. |
+| `topology.view.neighbours` | `neighbours` | Current neighbours: peers this node keeps alive and syncs with on its own. At least `k` (links chosen by either end count); well above `k` on a stable member that many others chose, bounded at about 8 × k. |
+| `topology.view.stableNeighbours` | `stable_neighbours` | Of those, stable members. |
+| `topology.view.fullMesh` | `full_mesh` | Every reachable member is a neighbour (at most `k + 1` reachable members). |
+
+`view` is absent until the node is ready and knows the member list; until then
+everyone counts as a neighbour.
+
+**Interpretation.** `members − reachable − 1` is how many members are offline or
+unreachable from here; if it is half or more of the electorate, no remake can commit.
+`neighbours` far above `k` on one node means it is a hub (a stable member among few),
+which is by design but costs it bandwidth.
+
+## Stability
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `stability.tier` | `stable` | What this node published about itself: `stable` once connected at least 95% of a full day, `transient` again below 80%. |
+| `stability.connectedFraction` | `connected_ratio` | Fraction of the observed time (up to 24 h) with at least one live peer. |
+| `stability.observedSecs` | `connectivity_observed_seconds` | How much time the fraction is based on; a day at most. |
+
+A node that restarted keeps a stable tier only if a peer says it was away less than an
+hour; otherwise it starts over with no history, and `observedSecs` says so.
+
+## Propagation
+
+How long changes written elsewhere took to reach this node, from the writer's commit
+stamp to the apply here, for every change in every applied sync message once caught up.
+The apply is timed by the peers' clocks: this node's own offset from them
+(`clock.offsetMs`, Clocks) is taken off.
+A histogram with buckets at 100 ms, 250 ms, 500 ms, 1 s, 2.5 s, 5 s, 10 s, 30 s, 1 min,
+5 min, 1 h, and the rest.
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `propagation.count`, `.sumMs` | `propagation_delay_seconds_count`, `_sum` | For the mean. |
+| `propagation.buckets[]` | `propagation_delay_seconds_bucket{le}` | Per bucket (not cumulative in JSON; cumulative in Prometheus, as histograms are). |
+| `propagation.ahead` | `propagation_clock_ahead_total` | Changes stamped later than their arrival (by the peers' clocks): the writer's clock is ahead. Counted in the first bucket. |
+
+**Interpretation.** The delay includes the writer's clock skew (this node's own is taken
+off), so changes from a writer with a badly set clock show nonsense here; `ahead`
+growing is the sign, and that writer's own `clock.off` ([`trust.md`](trust.md), "What depends on
+clocks"). Writers aren't known by node (Automerge's actors are per run), so their skew
+can't be taken off too. With sane clocks: in a full mesh most
+changes land in the first buckets; in a partial mesh each hop adds about a round trip
+plus the 200 ms announcement delay. Mass in the minutes buckets means changes arrive by
+keepalive-triggered pulls rather than relay, i.e. announcements are being lost.
+
+## Clocks
+
+Each PONG says when its sender answered, by its clock, so every peer's clock is known
+to within half a round trip (`clock` per peer, in Per peer). This node's own clock is
+the odd one out when it differs from most of its active peers' by more than
+`--clock-skew-warn` (2 s by default), for sure: beyond their bounds.
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `clock.peers` | `clock_peers_compared` | Active peers whose clocks were compared (a PONG from them lately). |
+| `clock.offsetMs` | `clock_offset_seconds` | How far this node's clock is ahead of its peers' (behind if negative): the median over them. Absent with none compared. |
+| `clock.behind`, `clock.ahead` | — | Peers whose clocks are behind ours, and ahead, by more than `--clock-skew-warn` for sure. |
+| `clock.off` | `clock_off` | Whether this node's clock is off: more than half of them differ from it beyond the limit, the same way. |
+| `clock.limitMs` | — | The limit (`--clock-skew-warn`). |
+
+**Interpretation.** A peer or two `ahead` or `behind` are their clocks, which they warn
+about themselves; `off` is ours. While it is, the log warns every 10 minutes, and says
+when it agrees again, and `/health` says so in a second line (still `200`). A clock set
+anew (by NTP, a resumed machine, by hand) by more than the limit shows as a jump in the
+log: this node's (a warning) when most peers' clocks moved alike, else the peers' whose
+did. Two nodes alone can't tell whose clock is off: both warn. Clocks matter for alive
+stamps, remake deadlines and the propagation figures ([`trust.md`](trust.md), "What depends on
+clocks"); setting them is the operating system's job (NTP).
+
+## Traffic
+
+Messages and payload bytes by direction (`sent`, `received`, `refused`) and kind
+(`hello`, `helloReply`, `sync`, `status`, `statusReply`, `blob`, `identity` for WHO and
+ME, counted under the address they went to or came from, so only in the totals, and
+`paths` for PING and PONG: per peer, but the paths layer's own PINGs under the address,
+so only in the totals). Payload bytes are what a transport's message carries (the wire
+encoding): not the transport's framing (for Mycelium, its topic and the base64 of its
+HTTP API). `refused` is what the transports wouldn't accept to send (no path to the
+peer), not counted as sent.
+
+| JSON | Prometheus | Scope |
+|---|---|---|
+| `traffic` in `/admin/stats` | `messages_total{direction,kind}`, `payload_bytes_total{direction,kind}` | All peers, including ones no longer listed. |
+| `traffic` per entry in `/peers` | `peer_messages_total{peer,direction,kind}`, `peer_payload_bytes_total{peer,…}` | One peer. |
+
+**Interpretation.** Keepalives (`paths` for PINGs and PONGs, every third one `status` + `statusReply`) should be about two per neighbour per
+keepalive interval; much more means announcements (relay) or heals. `sync` bytes
+dominate while catching up and on big writes; a steady `sync` stream on an idle swarm
+means two peers disagree and keep healing. `refused` growing for one peer means no
+transport has a path to it while we still consider it a neighbour.
+
+## Transports
+
+How the transports find and reach peers, under `transports` in `/admin/stats`; each
+part is absent (`null`) where it doesn't apply.
+
+The identity layer (`transport::identity`, [`peers.md`](peers.md)), under `transports.identity`:
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `proven` | `identity_proven_nodes{transport}` | Nodes proven at an address now, by transport; a node on several counts on each. |
+| `relayed` | `identity_relayed_nodes{transport}` | Of those, the nodes the transport reaches only through a relay (libp2p), by transport; in JSON absent where none is. Many means hole punching doesn't get through (`libp2p.holePunches`). |
+| `hinted` | `identity_hinted_nodes` | Nodes known only from the member list or HELLO_REPLYs, proven nowhere yet. |
+| `proofsRejected` | `identity_proofs_rejected_total` | WHO and ME that didn't hold: a bad signature, or made for other addresses. Normally 0; anything else is someone replaying or forging proofs. |
+| `addressChanges` | `identity_address_changes_total` | Nodes that proved themselves at another address on a transport than before: a restart with a new libp2p key or Mycelium key. Frequent ones mean a node doesn't keep its transport key (`--libp2p-key-file`, `--mycelium-key-file`). |
+| `droppedUnproven` | `identity_dropped_unproven_total` | Messages dropped because no node proved itself at their address (the address is asked WHO). Some after a restart are normal; a steady stream means something keeps sending from an address that never proves itself. |
+| `refusedUnproven` | `identity_refused_unproven_total` | Sends refused because the peer hadn't proven where it is yet; retried once it has. |
+| `seedsUnanswered` | `identity_seeds_unanswered_total` | Seeds at which no node answered WHO in time. |
+
+The paths layer (`transport::paths`, [`peers.md`](peers.md)), under `transports.paths`: the
+paths to peers, measured by PINGs.
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `paths[]` (`address`, `rttMs`, `down`, `connection`) | — | Every path measured: those of peers this node chose among or sent to in the last 5 minutes, and every path its keepalive PINGs went over (a peer's only path too). |
+| `probesSent`, `probesAnswered`, `probesUnanswered` | `paths_probes_total{result}` | PINGs, the keepalive's and every 30 s (or every keepalive interval, if that is longer) per path in use that no keepalive PING went over in that time (a busy pair needs no keepalives, so its paths would go unmeasured), and their PONGs. Unanswered ones (no PONG within 15 s) on one path mean it doesn't deliver; three in a row make it down. |
+| `sendFailures` | `paths_send_failures_total` | Sends that failed on a path; each is tried at once at the peer's next path. |
+| `failovers` | `paths_failovers_total` | Sends that went over another path after one failed. |
+| `switches` | `paths_switches_total` | Changes of the path in use for one faster by a quarter (and 5 ms). Frequent ones mean two paths about as fast, with changing delays. |
+
+Per peer, in `/peers` (`destination.paths`), and as `peer_path_rtt_seconds{peer,transport}`
+and `peer_path_down{peer,transport}`: each path to it, best first. A path's `address` is
+where it goes now: for libp2p, the remote address of its connection, ending in the peer ID
+(just the peer ID while there is none); a Mycelium public key; an onion service.
+
+libp2p (`--libp2p-enable`), under `transports.libp2p`:
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `reachability`, `publicAddress` | `libp2p_reachability{status}` | What AutoNAT says: `unknown`, `private` (others can't dial this node: it needs a relay) or `public`, with the address it confirmed. |
+| `connections.direct`, `.relayed` | `libp2p_connections{kind}` | Open connections now, direct or through a relay. |
+| `relaySlots.held` | `libp2p_relay_slots` | Slots held on relays now, to be reachable through them. |
+| `relaySlots.paused` | `libp2p_relays_paused` | Relays not picked again for a while: 10 minutes after refusing a slot, 30 seconds after not granting it otherwise. |
+| `relaySlots.granted`, `.lost`, `.notGranted` | `libp2p_relay_slot_events_total{event}` | Slots granted (renewals not counted), lost after being granted (not when one moves: reserved again at once over the connection kept, as a duplicate connection to the relay closed), and asked for but never granted. `notGranted` growing: a relay is unreachable, or another peer answers at its address (a relay restarted without its libp2p key). |
+| `holePunches.succeeded`, `.failed` | `libp2p_hole_punches_total{result}` | Direct connections tried over a relayed one (DCUtR). A failure leaves the relay carrying the traffic. |
+| `dialFailures` | `libp2p_dial_failures_total{cause}` | Connections that couldn't be made: `unreachable` (refused, timed out, no route), `wrongPeerId` (another peer at the address: one restarted without its libp2p key, or a NATed peer's LAN address in a range ours uses too; that address isn't taken for the peer again), `noAddresses`, `other`. |
+
+A relay that is no node (`--relay-only`) has `/admin/stats` too: its `peerId`, its
+`addresses` (as `libp2p:` addresses, ready for `_continuum/relays`), `uptimeSecs`, and
+`transports.libp2p` as above (Prometheus: `relay_info{peer_id}`, `uptime_seconds` and
+the `libp2p_…` series).
+
+**Interpretation.** A node that should be dialable but shows `private` has a port
+closed or not forwarded. `relayed` connections that stay while `holePunches.failed`
+grows means the relays carry the traffic for good: fine within their limits, but it
+costs the relay. `wrongPeerId` and `addressChanges` together point at a node restarted
+without its keys.
+
+## Incidents
+
+Counted per peer, summed in `/admin/stats` over the peers currently listed (a
+forgotten peer's are gone).
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `incidents.replyTimeouts` | `peer_reply_timeouts_total` | Our sync messages the peer didn't answer in time (10 s, 30 s with Tor enabled, doubling to 30 times that); resent. Lost messages, or a slow peer. |
+| `incidents.heals` | `peer_heals_total` | Re-handshakes because our heads and the peer's differed for `resync_after` with no sync between ([`peers.md`](peers.md), "Keepalive and self-healing"). Rare in a healthy swarm; frequent means sync state keeps getting out of step. |
+| `incidents.dormancies` | `peer_dormancies_total` | Times the peer went dormant (nothing received for `dormant_after`). |
+| `incidents.dormantMs` | `peer_dormant_seconds_total` | Time it spent dormant, up to its last return. |
+
+## Per peer
+
+In `/peers`, and as `peer_…` series with a `peer` label:
+
+| JSON | Prometheus | Meaning |
+|---|---|---|
+| `status`, `dormantForSecs` | `peer_unknown`, `peer_dormant` | Unknown (nothing heard from it since this node started: known from the member list or another peer), active, or dormant, and for how long. |
+| `lastSeenSecsAgo` | `peer_last_seen_seconds` | Since any message from it. |
+| `neighbour` | `peer_neighbour` | Whether we initiate towards it. |
+| `destination` | `peer_destination_info{transport,proven,connection}` | Where messages to it go: the address (its transport), whether it proved itself there (else sends wait for its proof), and the connection there: libp2p's `direct` or `relayed`, Tor's `direct` (an onion circuit); none on Mycelium; while `relayed` after a failed hole punch, `punching`: punches failed in a row (`failed`) and when it's tried again (`nextInSecs`). |
+| `destination.paths[].rttMs`, `.down`, `.connection` | `peer_path_rtt_seconds{transport}`, `peer_path_down{transport}` | Every path to it, best first: its round-trip time (smoothed, from PINGs), whether it is down, and the transport's connection there (`direct` or `relayed`, where it has connections). The first is the one in use: the peer's round-trip time. |
+| `lastSyncSecsAgo` | `peer_last_sync_seconds` | Since a sync message from it was applied (or the epoch changed). |
+| `divergedForSecs` | `peer_diverged_seconds` | How long its heads have differed from ours with no sync message from it in between; absent while they match. Long values precede a healing re-handshake ([`peers.md`](peers.md)). |
+| `syncThroughput` | `peer_sync_throughput_bytes_per_second` | Sync bytes per second over exchanges of at least 32 KiB, round trip taken off, smoothed. Absent before the first large exchange; comes with its age (`secsAgo`) since large exchanges are rare. This is the rate the peer delivers at in practice (its apply time included), not the path's capacity. |
+| `clock.offsetMs`, `.withinMs` | `peer_clock_offset_seconds`, `peer_clock_offset_within_seconds` | How far its clock is ahead of ours (behind if negative), give or take. Each PONG says when the peer answered, by its clock, which lies between our PING going and the PONG coming: within half the round trip of its middle, however unevenly the two ways took. Of its paths and the last 5 minutes, the narrowest bound is kept; a sample it contradicts (a clock set anew) replaces it at once. Absent before a PONG from it. |
+| `epoch`, `sameEpoch`, `heads` | — | What it last advertised; `sameEpoch: false` means we don't sync with it. |
+| `handshakeComplete`, `hasPendingDelivery` | — | Session state, for debugging. |
+
+## Disk
+
+Per kind of the node's own files (`state`, `document`, `backup`, `remake`), since
+start: `reads`, `readBytes`, `writes` (whole files), `writtenBytes`, `appends`,
+`appendedBytes`, `errors`. Prometheus: `disk_…_total{file}`. The document file is the
+only one written steadily: `appendedBytes` over time is the write rate it costs the
+device; a rising `writes` count there means rewrites because the appended part keeps
+outgrowing the full save (busy swarm, or a `--document-rewrite` set low).
+
+## The dashboard
+
+In builds with the `dashboard` cargo feature, `GET /admin/dashboard` is a page showing
+`/admin/stats`, `/peers` and a map of the swarm, live: [`dashboard.md`](dashboard.md), and how it works
+underneath (events, LINKS) at its end. Its LINKS are the one thing a node sends only to
+measure: the links every member reports, while someone looks.
+
+## What is not measured
+
+* Nothing about other nodes but what they send us: no probes, no remote figures (but
+  the dashboard's LINKS).
+* Clock skew between nodes that aren't each other's peers: only peers' clocks are
+  compared (`clock`, per peer).
+* Remake durations and outcomes: the remake reports (`GET /admin/remakes`) hold them.
+* Mycelium's own routing, retransmissions and queueing: the daemon's API has them.