path: root/docs/measurements.md

# Measurements

What a node measures about itself and its peers, where to read it, and what the
numbers mean. Everything here comes from traffic and work the node has anyway; it
sends no probes of its own and nothing is measured by anyone else, but for the
dashboard's map (LINKS, only while someone looks at it).

Four places to read:

* `GET /admin/stats`: the node's figures as JSON, or as Prometheus text when asked
  for it (`Accept: text/plain` or `application/openmetrics-text`). Prometheus names
  are `continuum_…`; per-peer series carry a `peer` label (the node ID) and can be
  joined with `continuum_peer_info` for the alias.
* `GET /peers`: one entry per known peer, keyed by its node ID, with the per-peer
  figures.
* The log: incidents as they happen (a peer going dormant, a heal, a stalled catch-up),
  at warn or info; `PUT /admin/log` changes the filter at runtime.
* `GET /admin/dashboard` (builds with the `dashboard` cargo feature): a page with the
  two above and a map of the swarm, live ([`dashboard.md`](dashboard.md)).

All counters count since the process started; a restart resets them. Durations are
whole seconds in JSON unless named `…Ms`; Prometheus has seconds throughout.

## About the node

| JSON | Prometheus | Meaning |
|---|---|---|
| `uptimeSecs` | `uptime_seconds` | Since the process started. |
| `ready` | `ready` | The node holds the swarm's document and serves writes ([`lifecycle.md`](lifecycle.md)). |
| `epoch` | `epoch_generation` | The local epoch; the generation counts remakes since genesis. |
| `staleHelloReplies` | `stale_hello_replies_total` | HELLO_REPLYs to a HELLO from an earlier run of this node, ignored. A few after every restart are normal (a transport queued them, Mycelium's above all); a steady stream means a peer keeps answering an old HELLO. |

## The document

| JSON | Prometheus | Meaning |
|---|---|---|
| `document.heads` | `document_heads` | Heads of the local document now. 1 means every change has been merged into one line of history; more means concurrent writes not yet followed by a merging write. |
| `document.maxHeads` | `document_max_heads` | The most heads seen right after applying a sync message. A high value shows how concurrent the swarm's writing gets; it also bounds keepalive size (above 32 heads, keepalives carry a digest and 32 of them). |
| `document.savedBytes` | `document_saved_bytes` | The document as Automerge saves it (compressed): roughly what a joining node downloads and what a document file rewrite writes. Measured in the background, only after a change, at least a minute apart and at most 1% of the time (a document that takes 2 s to save is measured every 200 s at most); an export (`GET /admin/document`) counts as a measurement. `savedBytesSecsAgo` and `savedBytesTookMs` say how fresh and how costly. |
| `document.lockWait.read`, `.write`: `count`, `sumUs`, `maxUs`, `buckets[]` (`leUs`) | `document_lock_wait_seconds{mode}` (histogram), `document_lock_wait_max_seconds{mode}` | How long takers of the document's lock waited for it, since start, by mode. Readers (the API's reads, topology, stats, generating sync messages, serving changes and deltas, saves) wait only for a writer; writers (applying sync messages and fetched changes, writes, epoch switches) wait for everyone. Buckets from 10 µs to 10 s, the last unbounded; not cumulative in JSON. |

**Interpretation.** `savedBytes` growing without bound is the reason to compact
([`remakes.md`](remakes.md)); `savedBytesTookMs` is a direct measure of what a switch, a rejoin and a
document file rewrite cost in CPU.

The node has one document, and nearly all work on it takes its lock, so `lockWait` says
whether more cores would help: waits of a few microseconds mean the lock is free when
asked for; write waits in the milliseconds (`sumUs` / `count`), growing with the number
of neighbours, mean applying one peer's sync messages queues behind the others'. A
high `maxUs` alone is a single long hold (a large sync message, a remake), not
contention.

## Catching up

| JSON | Prometheus | Meaning |
|---|---|---|
| `catchUp.sources` | `catch_up_sources_total` | Catch-up sources chosen. 1 is normal; more means the first source was given up (quiet, not ready, refused, or stalled) and another took over. |
| `catchUp.doneAfterSecs` | `catch_up_duration_seconds` | From start until the initial sync was done; absent until then, and on a node that was ready at once (genesis, document file, recovery). |
| `catchUp.syncBytes` | `catch_up_sync_bytes` | Sync payload received until then, from every peer: what joining cost. |

Compare `syncBytes` with `savedBytes`: much more received than the document is large
means the node got the document from several peers before the source rule took hold,
or caught up over many small messages.

## Remakes

| JSON | Prometheus | Meaning |
|---|---|---|
| `remakes.undecided` | `remakes_undecided` | Remakes of the local epoch not decided yet, expired ones included (`GET /admin/remakes`). |
| `remakes.committed` | `remakes_committed` | Committed remakes of the local epoch: 1 for the moment between a commit and the switch. More than one is a double commit, on which no node switches by itself: alert on it. |
| `remakes.rejoining` | `rejoining` | The node syncs a newer epoch from scratch ([`lifecycle.md`](lifecycle.md)). |
| `remakes.catchUpAttempts` | `catch_up_attempts` | Attempts at catching up with a newer epoch; 0 when not catching up. A growing count means the node can't get the epoch its peers advertise. |
| `remakes.quarantined`, `remakes.oldestQuarantineSecs` | `quarantined`, `oldest_quarantine_seconds` | Change sets set aside on a rejoin (`GET /admin/quarantine`). They live in memory only, so a restart loses them: alerting is how to learn of them. |
| `remakes.archivedEpochs` | `archived_epochs` | Old epochs' documents kept in memory (`archiveRetention`). |

## Sync and relay

| JSON | Prometheus | Meaning |
|---|---|---|
| `sync.lastFromNeighbourSecs` | `last_sync_from_neighbour_seconds` | Since a sync message from any neighbour was applied. Growing without bound while others write: the node is cut off from the swarm's changes. |
| `sync.divergedPeers` | `diverged_peers` | Peers whose heads differ from ours with no sync message in between. |
| `sync.largestReceivedBytes` | `largest_sync_received_bytes` | The largest sync message received since start. Larger transfers come as deltas (blobs of 16 chunks or more, [`peers.md`](peers.md)), so this stays below 16 × `--blob-chunk` from nodes of this version. |
| `relay.announceRounds` | `announce_rounds_total` | Announcement rounds: our new heads to the neighbours after applying their changes (partial mesh only). |
| `relay.pulls` | `pulls_total` | Pulls of changes we lacked from a neighbour that announced them. |
| `relay.pullsQueued` | `pulls_queued_total` | Announcements of heads we lacked that waited for the pull in progress. Many compared with `pulls`: changes arrive from several sides at once. |
| `relay.blobRequestsDropped` | `blob_requests_dropped_total` | Blob requests not answered because too many were being answered; the requester asks again or asks another peer. Also logged as a warning, at most once a minute with the count. |

## Topology

| JSON | Prometheus | Meaning |
|---|---|---|
| `topology.k` | `neighbours_target` | `--neighbours`. |
| `topology.view.members` | `members` | Registered members, this node included. |
| `topology.view.reachable` | `reachable_members` | Members other than this node not judged unreachable (three unanswered HELLOs, or dormant). |
| `topology.view.stable` | `stable_members` | Members whose registry entry says `stable`. |
| `topology.view.neighbours` | `neighbours` | Current neighbours: peers this node keeps alive and syncs with on its own. At least `k` (links chosen by either end count); well above `k` on a stable member that many others chose, bounded at about 8 × k. |
| `topology.view.stableNeighbours` | `stable_neighbours` | Of those, stable members. |
| `topology.view.fullMesh` | `full_mesh` | Every reachable member is a neighbour (at most `k + 1` reachable members). |

`view` is absent until the node is ready and knows the member list; until then
everyone counts as a neighbour.

**Interpretation.** `members − reachable − 1` is how many members are offline or
unreachable from here; if it is half or more of the electorate, no remake can commit.
`neighbours` far above `k` on one node means it is a hub (a stable member among few),
which is by design but costs it bandwidth.

## Stability

| JSON | Prometheus | Meaning |
|---|---|---|
| `stability.tier` | `stable` | What this node published about itself: `stable` once connected at least 95% of a full day, `transient` again below 80%. |
| `stability.connectedFraction` | `connected_ratio` | Fraction of the observed time (up to 24 h) with at least one live peer. |
| `stability.observedSecs` | `connectivity_observed_seconds` | How much time the fraction is based on; a day at most. |

A node that restarted keeps a stable tier only if a peer says it was away less than an
hour; otherwise it starts over with no history, and `observedSecs` says so.

## Propagation

How long changes written elsewhere took to reach this node, from the writer's commit
stamp to the apply here, for every change in every applied sync message once caught up.
The apply is timed by the peers' clocks: this node's own offset from them
(`clock.offsetMs`, Clocks) is taken off.
A histogram with buckets at 100 ms, 250 ms, 500 ms, 1 s, 2.5 s, 5 s, 10 s, 30 s, 1 min,
5 min, 1 h, and the rest.

| JSON | Prometheus | Meaning |
|---|---|---|
| `propagation.count`, `.sumMs` | `propagation_delay_seconds_count`, `_sum` | For the mean. |
| `propagation.buckets[]` | `propagation_delay_seconds_bucket{le}` | Per bucket (not cumulative in JSON; cumulative in Prometheus, as histograms are). |
| `propagation.ahead` | `propagation_clock_ahead_total` | Changes stamped later than their arrival (by the peers' clocks): the writer's clock is ahead. Counted in the first bucket. |

**Interpretation.** The delay includes the writer's clock skew (this node's own is taken
off), so changes from a writer with a badly set clock show nonsense here; `ahead`
growing is the sign, and that writer's own `clock.off` ([`trust.md`](trust.md), "What depends on
clocks"). Writers aren't known by node (Automerge's actors are per run), so their skew
can't be taken off too. With sane clocks: in a full mesh most
changes land in the first buckets; in a partial mesh each hop adds about a round trip
plus the 200 ms announcement delay. Mass in the minutes buckets means changes arrive by
keepalive-triggered pulls rather than relay, i.e. announcements are being lost.

## Clocks

Each PONG says when its sender answered, by its clock, so every peer's clock is known
to within half a round trip (`clock` per peer, in Per peer). This node's own clock is
the odd one out when it differs from most of its active peers' by more than
`--clock-skew-warn` (2 s by default), for sure: beyond their bounds.

| JSON | Prometheus | Meaning |
|---|---|---|
| `clock.peers` | `clock_peers_compared` | Active peers whose clocks were compared (a PONG from them lately). |
| `clock.offsetMs` | `clock_offset_seconds` | How far this node's clock is ahead of its peers' (behind if negative): the median over them. Absent with none compared. |
| `clock.behind`, `clock.ahead` | — | Peers whose clocks are behind ours, and ahead, by more than `--clock-skew-warn` for sure. |
| `clock.off` | `clock_off` | Whether this node's clock is off: more than half of them differ from it beyond the limit, the same way. |
| `clock.limitMs` | — | The limit (`--clock-skew-warn`). |

**Interpretation.** A peer or two `ahead` or `behind` are their clocks, which they warn
about themselves; `off` is ours. While it is, the log warns every 10 minutes, and says
when it agrees again, and `/health` says so in a second line (still `200`). A clock set
anew (by NTP, a resumed machine, by hand) by more than the limit shows as a jump in the
log: this node's (a warning) when most peers' clocks moved alike, else the peers' whose
did. Two nodes alone can't tell whose clock is off: both warn. Clocks matter for alive
stamps, remake deadlines and the propagation figures ([`trust.md`](trust.md), "What depends on
clocks"); setting them is the operating system's job (NTP).

## Traffic

Messages and payload bytes by direction (`sent`, `received`, `refused`) and kind
(`hello`, `helloReply`, `sync`, `status`, `statusReply`, `blob`, `identity` for WHO and
ME, counted under the address they went to or came from, so only in the totals, and
`paths` for PING and PONG: per peer, but the paths layer's own PINGs under the address,
so only in the totals). Payload bytes are what a transport's message carries (the wire
encoding): not the transport's framing (for Mycelium, its topic and the base64 of its
HTTP API). `refused` is what the transports wouldn't accept to send (no path to the
peer), not counted as sent.

| JSON | Prometheus | Scope |
|---|---|---|
| `traffic` in `/admin/stats` | `messages_total{direction,kind}`, `payload_bytes_total{direction,kind}` | All peers, including ones no longer listed. |
| `traffic` per entry in `/peers` | `peer_messages_total{peer,direction,kind}`, `peer_payload_bytes_total{peer,…}` | One peer. |

**Interpretation.** Keepalives (`paths` for PINGs and PONGs, every third one `status` + `statusReply`) should be about two per neighbour per
keepalive interval; much more means announcements (relay) or heals. `sync` bytes
dominate while catching up and on big writes; a steady `sync` stream on an idle swarm
means two peers disagree and keep healing. `refused` growing for one peer means no
transport has a path to it while we still consider it a neighbour.

## Transports

How the transports find and reach peers, under `transports` in `/admin/stats`; each
part is absent (`null`) where it doesn't apply.

The identity layer (`transport::identity`, [`peers.md`](peers.md)), under `transports.identity`:

| JSON | Prometheus | Meaning |
|---|---|---|
| `proven` | `identity_proven_nodes{transport}` | Nodes proven at an address now, by transport; a node on several counts on each. |
| `relayed` | `identity_relayed_nodes{transport}` | Of those, the nodes the transport reaches only through a relay (libp2p), by transport; in JSON absent where none is. Many means hole punching doesn't get through (`libp2p.holePunches`). |
| `hinted` | `identity_hinted_nodes` | Nodes known only from the member list or HELLO_REPLYs, proven nowhere yet. |
| `proofsRejected` | `identity_proofs_rejected_total` | WHO and ME that didn't hold: a bad signature, or made for other addresses. Normally 0; anything else is someone replaying or forging proofs. |
| `addressChanges` | `identity_address_changes_total` | Nodes that proved themselves at another address on a transport than before: a restart with a new libp2p key or Mycelium key. Frequent ones mean a node doesn't keep its transport key (`--libp2p-key-file`, `--mycelium-key-file`). |
| `droppedUnproven` | `identity_dropped_unproven_total` | Messages dropped because no node proved itself at their address (the address is asked WHO). Some after a restart are normal; a steady stream means something keeps sending from an address that never proves itself. |
| `refusedUnproven` | `identity_refused_unproven_total` | Sends refused because the peer hadn't proven where it is yet; retried once it has. |
| `seedsUnanswered` | `identity_seeds_unanswered_total` | Seeds at which no node answered WHO in time. |

The paths layer (`transport::paths`, [`peers.md`](peers.md)), under `transports.paths`: the
paths to peers, measured by PINGs.

| JSON | Prometheus | Meaning |
|---|---|---|
| `paths[]` (`address`, `rttMs`, `down`, `connection`) | — | Every path measured: those of peers this node chose among or sent to in the last 5 minutes, and every path its keepalive PINGs went over (a peer's only path too). |
| `probesSent`, `probesAnswered`, `probesUnanswered` | `paths_probes_total{result}` | PINGs, the keepalive's and every 30 s (or every keepalive interval, if that is longer) per path in use that no keepalive PING went over in that time (a busy pair needs no keepalives, so its paths would go unmeasured), and their PONGs. Unanswered ones (no PONG within 15 s) on one path mean it doesn't deliver; three in a row make it down. |
| `sendFailures` | `paths_send_failures_total` | Sends that failed on a path; each is tried at once at the peer's next path. |
| `failovers` | `paths_failovers_total` | Sends that went over another path after one failed. |
| `switches` | `paths_switches_total` | Changes of the path in use for one faster by a quarter (and 5 ms). Frequent ones mean two paths about as fast, with changing delays. |

Per peer, in `/peers` (`destination.paths`), and as `peer_path_rtt_seconds{peer,transport}`
and `peer_path_down{peer,transport}`: each path to it, best first. A path's `address` is
where it goes now: for libp2p, the remote address of its connection, ending in the peer ID
(just the peer ID while there is none); a Mycelium public key; an onion service.

libp2p (`--libp2p-enable`), under `transports.libp2p`:

| JSON | Prometheus | Meaning |
|---|---|---|
| `reachability`, `publicAddress` | `libp2p_reachability{status}` | What AutoNAT says: `unknown`, `private` (others can't dial this node: it needs a relay) or `public`, with the address it confirmed. |
| `connections.direct`, `.relayed` | `libp2p_connections{kind}` | Open connections now, direct or through a relay. |
| `relaySlots.held` | `libp2p_relay_slots` | Slots held on relays now, to be reachable through them. |
| `relaySlots.paused` | `libp2p_relays_paused` | Relays not picked again for a while: 10 minutes after refusing a slot, 30 seconds after not granting it otherwise. |
| `relaySlots.granted`, `.lost`, `.notGranted` | `libp2p_relay_slot_events_total{event}` | Slots granted (renewals not counted), lost after being granted (not when one moves: reserved again at once over the connection kept, as a duplicate connection to the relay closed), and asked for but never granted. `notGranted` growing: a relay is unreachable, or another peer answers at its address (a relay restarted without its libp2p key). |
| `holePunches.succeeded`, `.failed` | `libp2p_hole_punches_total{result}` | Direct connections tried over a relayed one (DCUtR). A failure leaves the relay carrying the traffic. |
| `dialFailures` | `libp2p_dial_failures_total{cause}` | Connections that couldn't be made: `unreachable` (refused, timed out, no route), `wrongPeerId` (another peer at the address: one restarted without its libp2p key, or a NATed peer's LAN address in a range ours uses too; that address isn't taken for the peer again), `noAddresses`, `other`. |

A relay that is no node (`--relay-only`) has `/admin/stats` too: its `peerId`, its
`addresses` (as `libp2p:` addresses, ready for `_continuum/relays`), `uptimeSecs`, and
`transports.libp2p` as above (Prometheus: `relay_info{peer_id}`, `uptime_seconds` and
the `libp2p_…` series).

**Interpretation.** A node that should be dialable but shows `private` has a port
closed or not forwarded. `relayed` connections that stay while `holePunches.failed`
grows means the relays carry the traffic for good: fine within their limits, but it
costs the relay. `wrongPeerId` and `addressChanges` together point at a node restarted
without its keys.

## Incidents

Counted per peer, summed in `/admin/stats` over the peers currently listed (a
forgotten peer's are gone).

| JSON | Prometheus | Meaning |
|---|---|---|
| `incidents.replyTimeouts` | `peer_reply_timeouts_total` | Our sync messages the peer didn't answer in time (10 s, 30 s with Tor enabled, doubling to 30 times that); resent. Lost messages, or a slow peer. |
| `incidents.heals` | `peer_heals_total` | Re-handshakes because our heads and the peer's differed for `resync_after` with no sync between ([`peers.md`](peers.md), "Keepalive and self-healing"). Rare in a healthy swarm; frequent means sync state keeps getting out of step. |
| `incidents.dormancies` | `peer_dormancies_total` | Times the peer went dormant (nothing received for `dormant_after`). |
| `incidents.dormantMs` | `peer_dormant_seconds_total` | Time it spent dormant, up to its last return. |

## Per peer

In `/peers`, and as `peer_…` series with a `peer` label:

| JSON | Prometheus | Meaning |
|---|---|---|
| `status`, `dormantForSecs` | `peer_unknown`, `peer_dormant` | Unknown (nothing heard from it since this node started: known from the member list or another peer), active, or dormant, and for how long. |
| `lastSeenSecsAgo` | `peer_last_seen_seconds` | Since any message from it. |
| `neighbour` | `peer_neighbour` | Whether we initiate towards it. |
| `destination` | `peer_destination_info{transport,proven,connection}` | Where messages to it go: the address (its transport), whether it proved itself there (else sends wait for its proof), and the connection there: libp2p's `direct` or `relayed`, Tor's `direct` (an onion circuit); none on Mycelium; while `relayed` after a failed hole punch, `punching`: punches failed in a row (`failed`) and when it's tried again (`nextInSecs`). |
| `destination.paths[].rttMs`, `.down`, `.connection` | `peer_path_rtt_seconds{transport}`, `peer_path_down{transport}` | Every path to it, best first: its round-trip time (smoothed, from PINGs), whether it is down, and the transport's connection there (`direct` or `relayed`, where it has connections). The first is the one in use: the peer's round-trip time. |
| `lastSyncSecsAgo` | `peer_last_sync_seconds` | Since a sync message from it was applied (or the epoch changed). |
| `divergedForSecs` | `peer_diverged_seconds` | How long its heads have differed from ours with no sync message from it in between; absent while they match. Long values precede a healing re-handshake ([`peers.md`](peers.md)). |
| `syncThroughput` | `peer_sync_throughput_bytes_per_second` | Sync bytes per second over exchanges of at least 32 KiB, round trip taken off, smoothed. Absent before the first large exchange; comes with its age (`secsAgo`) since large exchanges are rare. This is the rate the peer delivers at in practice (its apply time included), not the path's capacity. |
| `clock.offsetMs`, `.withinMs` | `peer_clock_offset_seconds`, `peer_clock_offset_within_seconds` | How far its clock is ahead of ours (behind if negative), give or take. Each PONG says when the peer answered, by its clock, which lies between our PING going and the PONG coming: within half the round trip of its middle, however unevenly the two ways took. Of its paths and the last 5 minutes, the narrowest bound is kept; a sample it contradicts (a clock set anew) replaces it at once. Absent before a PONG from it. |
| `epoch`, `sameEpoch`, `heads` | — | What it last advertised; `sameEpoch: false` means we don't sync with it. |
| `handshakeComplete`, `hasPendingDelivery` | — | Session state, for debugging. |

## Disk

Per kind of the node's own files (`state`, `document`, `backup`, `remake`), since
start: `reads`, `readBytes`, `writes` (whole files), `writtenBytes`, `appends`,
`appendedBytes`, `errors`. Prometheus: `disk_…_total{file}`. The document file is the
only one written steadily: `appendedBytes` over time is the write rate it costs the
device; a rising `writes` count there means rewrites because the appended part keeps
outgrowing the full save (busy swarm, or a `--document-rewrite` set low).

## The dashboard

In builds with the `dashboard` cargo feature, `GET /admin/dashboard` is a page showing
`/admin/stats`, `/peers` and a map of the swarm, live: [`dashboard.md`](dashboard.md), and how it works
underneath (events, LINKS) at its end. Its LINKS are the one thing a node sends only to
measure: the links every member reports, while someone looks.

## What is not measured

* Nothing about other nodes but what they send us: no probes, no remote figures (but
  the dashboard's LINKS).
* Clock skew between nodes that aren't each other's peers: only peers' clocks are
  compared (`clock`, per peer).
* Remake durations and outcomes: the remake reports (`GET /admin/remakes`) hold them.
* Mycelium's own routing, retransmissions and queueing: the daemon's API has them.