Measurements

What a node measures about itself and its peers, where to read it, and what the numbers mean. Everything here comes from traffic and work the node has anyway; it sends no probes of its own and nothing is measured by anyone else, but for the dashboard's map (LINKS, only while someone looks at it).

Four places to read:

• GET /admin/stats: the node's figures as JSON, or as Prometheus text when asked for it (Accept: text/plain or application/openmetrics-text). Prometheus names are continuum_…; per-peer series carry a peer label (the node ID) and can be joined with continuum_peer_info for the alias.
• GET /peers: one entry per known peer, keyed by its node ID, with the per-peer figures.
• The log: incidents as they happen (a peer going dormant, a heal, a stalled catch-up), at warn or info; PUT /admin/log changes the filter at runtime.
• GET /admin/dashboard (builds with the dashboard cargo feature): a page with the two above and a map of the swarm, live (dashboard.md).

All counters count since the process started; a restart resets them. Durations are whole seconds in JSON unless named …Ms; Prometheus has seconds throughout.

About the node

JSON	Prometheus	Meaning
uptimeSecs	uptime_seconds	Since the process started.
ready	ready	The node holds the swarm's document and serves writes (lifecycle.md).
epoch	epoch_generation	The local epoch; the generation counts remakes since genesis.
staleHelloReplies	stale_hello_replies_total	HELLO_REPLYs to a HELLO from an earlier run of this node, ignored. A few after every restart are normal (a transport queued them, Mycelium's above all); a steady stream means a peer keeps answering an old HELLO.

The document

JSON	Prometheus	Meaning
document.heads	document_heads	Heads of the local document now. 1 means every change has been merged into one line of history; more means concurrent writes not yet followed by a merging write.
document.maxHeads	document_max_heads	The most heads seen right after applying a sync message. A high value shows how concurrent the swarm's writing gets; it also bounds keepalive size (above 32 heads, keepalives carry a digest and 32 of them).
document.savedBytes	document_saved_bytes	The document as Automerge saves it (compressed): roughly what a joining node downloads and what a document file rewrite writes. Measured in the background, only after a change, at least a minute apart and at most 1% of the time (a document that takes 2 s to save is measured every 200 s at most); an export (GET /admin/document) counts as a measurement. savedBytesSecsAgo and savedBytesTookMs say how fresh and how costly.
document.lockWait.read, .write: count, sumUs, maxUs, buckets[] (leUs)	document_lock_wait_seconds{mode} (histogram), document_lock_wait_max_seconds{mode}	How long takers of the document's lock waited for it, since start, by mode. Readers (the API's reads, topology, stats, generating sync messages, serving changes and deltas, saves) wait only for a writer; writers (applying sync messages and fetched changes, writes, epoch switches) wait for everyone. Buckets from 10 µs to 10 s, the last unbounded; not cumulative in JSON.

Interpretation. savedBytes growing without bound is the reason to compact (remakes.md); savedBytesTookMs is a direct measure of what a switch, a rejoin and a document file rewrite cost in CPU.

The node has one document, and nearly all work on it takes its lock, so lockWait says whether more cores would help: waits of a few microseconds mean the lock is free when asked for; write waits in the milliseconds (sumUs / count), growing with the number of neighbours, mean applying one peer's sync messages queues behind the others'. A high maxUs alone is a single long hold (a large sync message, a remake), not contention.

Catching up

JSON	Prometheus	Meaning
catchUp.sources	catch_up_sources_total	Catch-up sources chosen. 1 is normal; more means the first source was given up (quiet, not ready, refused, or stalled) and another took over.
catchUp.doneAfterSecs	catch_up_duration_seconds	From start until the initial sync was done; absent until then, and on a node that was ready at once (genesis, document file, recovery).
catchUp.syncBytes	catch_up_sync_bytes	Sync payload received until then, from every peer: what joining cost.

Compare syncBytes with savedBytes: much more received than the document is large means the node got the document from several peers before the source rule took hold, or caught up over many small messages.

Remakes

JSON	Prometheus	Meaning
remakes.undecided	remakes_undecided	Remakes of the local epoch not decided yet, expired ones included (GET /admin/remakes).
remakes.committed	remakes_committed	Committed remakes of the local epoch: 1 for the moment between a commit and the switch. More than one is a double commit, on which no node switches by itself: alert on it.
remakes.rejoining	rejoining	The node syncs a newer epoch from scratch (lifecycle.md).
remakes.catchUpAttempts	catch_up_attempts	Attempts at catching up with a newer epoch; 0 when not catching up. A growing count means the node can't get the epoch its peers advertise.
remakes.quarantined, remakes.oldestQuarantineSecs	quarantined, oldest_quarantine_seconds	Change sets set aside on a rejoin (GET /admin/quarantine). They live in memory only, so a restart loses them: alerting is how to learn of them.
remakes.archivedEpochs	archived_epochs	Old epochs' documents kept in memory (archiveRetention).

Sync and relay

JSON	Prometheus	Meaning
sync.lastFromNeighbourSecs	last_sync_from_neighbour_seconds	Since a sync message from any neighbour was applied. Growing without bound while others write: the node is cut off from the swarm's changes.
sync.divergedPeers	diverged_peers	Peers whose heads differ from ours with no sync message in between.
sync.largestReceivedBytes	largest_sync_received_bytes	The largest sync message received since start. Larger transfers come as deltas (blobs of 16 chunks or more, peers.md), so this stays below 16 × --blob-chunk from nodes of this version.
relay.announceRounds	announce_rounds_total	Announcement rounds: our new heads to the neighbours after applying their changes (partial mesh only).
relay.pulls	pulls_total	Pulls of changes we lacked from a neighbour that announced them.
relay.pullsQueued	pulls_queued_total	Announcements of heads we lacked that waited for the pull in progress. Many compared with pulls: changes arrive from several sides at once.
relay.blobRequestsDropped	blob_requests_dropped_total	Blob requests not answered because too many were being answered; the requester asks again or asks another peer. Also logged as a warning, at most once a minute with the count.

Topology

JSON	Prometheus	Meaning
topology.k	neighbours_target	--neighbours.
topology.view.members	members	Registered members, this node included.
topology.view.reachable	reachable_members	Members other than this node not judged unreachable (three unanswered HELLOs, or dormant).
topology.view.stable	stable_members	Members whose registry entry says stable.
topology.view.neighbours	neighbours	Current neighbours: peers this node keeps alive and syncs with on its own. At least k (links chosen by either end count); well above k on a stable member that many others chose, bounded at about 8 × k.
topology.view.stableNeighbours	stable_neighbours	Of those, stable members.
topology.view.fullMesh	full_mesh	Every reachable member is a neighbour (at most k + 1 reachable members).

view is absent until the node is ready and knows the member list; until then everyone counts as a neighbour.

Interpretation. members − reachable − 1 is how many members are offline or unreachable from here; if it is half or more of the electorate, no remake can commit. neighbours far above k on one node means it is a hub (a stable member among few), which is by design but costs it bandwidth.

Stability

JSON	Prometheus	Meaning
stability.tier	stable	What this node published about itself: stable once connected at least 95% of a full day, transient again below 80%.
stability.connectedFraction	connected_ratio	Fraction of the observed time (up to 24 h) with at least one live peer.
stability.observedSecs	connectivity_observed_seconds	How much time the fraction is based on; a day at most.

A node that restarted keeps a stable tier only if a peer says it was away less than an hour; otherwise it starts over with no history, and observedSecs says so.

Propagation

How long changes written elsewhere took to reach this node, from the writer's commit stamp to the apply here, for every change in every applied sync message once caught up. The apply is timed by the peers' clocks: this node's own offset from them (clock.offsetMs, Clocks) is taken off. A histogram with buckets at 100 ms, 250 ms, 500 ms, 1 s, 2.5 s, 5 s, 10 s, 30 s, 1 min, 5 min, 1 h, and the rest.

JSON	Prometheus	Meaning
propagation.count, .sumMs	propagation_delay_seconds_count, _sum	For the mean.
propagation.buckets[]	propagation_delay_seconds_bucket{le}	Per bucket (not cumulative in JSON; cumulative in Prometheus, as histograms are).
propagation.ahead	propagation_clock_ahead_total	Changes stamped later than their arrival (by the peers' clocks): the writer's clock is ahead. Counted in the first bucket.

Interpretation. The delay includes the writer's clock skew (this node's own is taken off), so changes from a writer with a badly set clock show nonsense here; ahead growing is the sign, and that writer's own clock.off (trust.md, "What depends on clocks"). Writers aren't known by node (Automerge's actors are per run), so their skew can't be taken off too. With sane clocks: in a full mesh most changes land in the first buckets; in a partial mesh each hop adds about a round trip plus the 200 ms announcement delay. Mass in the minutes buckets means changes arrive by keepalive-triggered pulls rather than relay, i.e. announcements are being lost.

Clocks

Each PONG says when its sender answered, by its clock, so every peer's clock is known to within half a round trip (clock per peer, in Per peer). This node's own clock is the odd one out when it differs from most of its active peers' by more than --clock-skew-warn (2 s by default), for sure: beyond their bounds.

JSON	Prometheus	Meaning
clock.peers	clock_peers_compared	Active peers whose clocks were compared (a PONG from them lately).
clock.offsetMs	clock_offset_seconds	How far this node's clock is ahead of its peers' (behind if negative): the median over them. Absent with none compared.
clock.behind, clock.ahead	—	Peers whose clocks are behind ours, and ahead, by more than --clock-skew-warn for sure.
clock.off	clock_off	Whether this node's clock is off: more than half of them differ from it beyond the limit, the same way.
clock.limitMs	—	The limit (--clock-skew-warn).

Interpretation. A peer or two ahead or behind are their clocks, which they warn about themselves; off is ours. While it is, the log warns every 10 minutes, and says when it agrees again, and /health says so in a second line (still 200). A clock set anew (by NTP, a resumed machine, by hand) by more than the limit shows as a jump in the log: this node's (a warning) when most peers' clocks moved alike, else the peers' whose did. Two nodes alone can't tell whose clock is off: both warn. Clocks matter for alive stamps, remake deadlines and the propagation figures (trust.md, "What depends on clocks"); setting them is the operating system's job (NTP).

Traffic

Messages and payload bytes by direction (sent, received, refused) and kind (hello, helloReply, sync, status, statusReply, blob, identity for WHO and ME, and paths for PING and PONG). The layers below the node count WHO and ME, and the paths layer's own PINGs and PONGs, under the address they went to or came from; a peer (/peers) has those of its addresses too, where it is known to be at them. Payload bytes are what a transport's message carries (the wire encoding): not the transport's framing (for Mycelium, its topic and the base64 of its HTTP API). refused is what the transports wouldn't accept to send (no path to the peer), not counted as sent.

JSON	Prometheus	Scope
traffic in /admin/stats	messages_total{direction,kind}, payload_bytes_total{direction,kind}	All peers, including ones no longer listed.
traffic per entry in /peers	peer_messages_total{peer,direction,kind}, peer_payload_bytes_total{peer,…}	One peer.

Interpretation. Keepalives (paths for PINGs and PONGs, every third one status + statusReply) should be about two per neighbour per keepalive interval; much more means announcements (relay) or heals. sync bytes dominate while catching up and on big writes; a steady sync stream on an idle swarm means two peers disagree and keep healing. refused growing for one peer means no transport has a path to it while we still consider it a neighbour.

Transports

How the transports find and reach peers, under transports in /admin/stats; each part is absent (null) where it doesn't apply.

The identity layer (transport::identity, peers.md), under transports.identity:

JSON	Prometheus	Meaning
proven	identity_proven_nodes{transport}	Nodes proven at an address now, by transport; a node on several counts on each.
relayed	identity_relayed_nodes{transport}	Of those, the nodes the transport reaches only through a relay (libp2p), by transport; in JSON absent where none is. Many means hole punching doesn't get through (libp2p.holePunches).
hinted	identity_hinted_nodes	Nodes known only from the member list or HELLO_REPLYs, proven nowhere yet.
proofsRejected	identity_proofs_rejected_total	WHO and ME that didn't hold: a bad signature, or made for other addresses. Normally 0; anything else is someone replaying or forging proofs.
addressChanges	identity_address_changes_total	Nodes that proved themselves at another address on a transport than before: a restart with a new libp2p key or Mycelium key. Frequent ones mean a node doesn't keep its transport key (--libp2p-key-file, --mycelium-key-file).
droppedUnproven	identity_dropped_unproven_total	Messages dropped because no node proved itself at their address (the address is asked WHO). Some after a restart are normal; a steady stream means something keeps sending from an address that never proves itself.
refusedUnproven	identity_refused_unproven_total	Sends refused because the peer hadn't proven where it is yet; retried once it has.
seedsUnanswered	identity_seeds_unanswered_total	Seeds at which no node answered WHO in time.

The paths layer (transport::paths, peers.md), under transports.paths: the paths to peers, measured by PINGs.

JSON	Prometheus	Meaning
paths[] (address, rttMs, down, connection)	—	Every path measured: those of peers this node chose among or sent to in the last 5 minutes, and every path its keepalive PINGs went over (a peer's only path too).
probesSent, probesAnswered, probesUnanswered	paths_probes_total{result}	PINGs, the keepalive's and every 30 s (or every keepalive interval, if that is longer) per path in use that no keepalive PING went over in that time (a busy pair needs no keepalives, so its paths would go unmeasured), and their PONGs. Unanswered ones (no PONG within 15 s) on one path mean it doesn't deliver; three in a row make it down.
sendFailures	paths_send_failures_total	Sends that failed on a path; each is tried at once at the peer's next path.
failovers	paths_failovers_total	Sends that went over another path after one failed.
switches	paths_switches_total	Changes of the path in use for one faster by a quarter (and 5 ms). Frequent ones mean two paths about as fast, with changing delays.

Per peer, in /peers (destination.paths), and as peer_path_rtt_seconds{peer,transport} and peer_path_down{peer,transport}: each path to it, best first. A path's address is where it goes now: for libp2p, the remote address of its connection, ending in the peer ID (just the peer ID while there is none); a Mycelium public key; an onion service.

libp2p (--libp2p-enable), under transports.libp2p:

JSON	Prometheus	Meaning
reachability, publicAddress	libp2p_reachability{status}	What AutoNAT says: unknown, private (others can't dial this node: it needs a relay) or public, with the address it confirmed.
connections.direct, .relayed	libp2p_connections{kind}	Open connections now, direct or through a relay.
relaySlots.held	libp2p_relay_slots	Slots held on relays now, to be reachable through them.
relaySlots.paused	libp2p_relays_paused	Relays not picked again for a while: 10 minutes after refusing a slot, 30 seconds after not granting it otherwise.
relaySlots.granted, .lost, .notGranted	libp2p_relay_slot_events_total{event}	Slots granted (renewals not counted), lost after being granted (not when one moves: reserved again at once over the connection kept, as a duplicate connection to the relay closed), and asked for but never granted. notGranted growing: a relay is unreachable, or another peer answers at its address (a relay restarted without its libp2p key).
holePunches.succeeded, .failed	libp2p_hole_punches_total{result}	Direct connections tried over a relayed one (DCUtR). A failure leaves the relay carrying the traffic.
dialFailures	libp2p_dial_failures_total{cause}	Connections that couldn't be made: unreachable (refused, timed out, no route), wrongPeerId (another peer at the address: one restarted without its libp2p key, or a NATed peer's LAN address in a range ours uses too; that address isn't taken for the peer again), noAddresses, other.

A relay that is no node (--relay-only) has /admin/stats too: its peerId, its addresses (as libp2p: addresses, ready for _continuum/relays), uptimeSecs, and transports.libp2p as above (Prometheus: relay_info{peer_id}, uptime_seconds and the libp2p_… series).

Interpretation. A node that should be dialable but shows private has a port closed or not forwarded. relayed connections that stay while holePunches.failed grows means the relays carry the traffic for good: fine within their limits, but it costs the relay. wrongPeerId and addressChanges together point at a node restarted without its keys.

Incidents

Counted per peer, summed in /admin/stats over the peers currently listed (a forgotten peer's are gone).

JSON	Prometheus	Meaning
incidents.replyTimeouts	peer_reply_timeouts_total	Our sync messages the peer didn't answer in time (10 s, 30 s with Tor enabled, doubling to 30 times that); resent. Lost messages, or a slow peer.
incidents.heals	peer_heals_total	Re-handshakes because our heads and the peer's differed for resync_after with no sync between (peers.md, "Keepalive and self-healing"). Rare in a healthy swarm; frequent means sync state keeps getting out of step.
incidents.dormancies	peer_dormancies_total	Times the peer went dormant (nothing received for dormant_after).
incidents.dormantMs	peer_dormant_seconds_total	Time it spent dormant, up to its last return.

Per peer

In /peers, and as peer_… series with a peer label:

JSON	Prometheus	Meaning
status, dormantForSecs	peer_unknown, peer_dormant	Unknown (nothing heard from it since this node started: known from the member list or another peer), active, or dormant, and for how long.
lastSeenSecsAgo	peer_last_seen_seconds	Since any message from it.
neighbour	peer_neighbour	Whether we initiate towards it.
destination	peer_destination_info{transport,proven,connection}	Where messages to it go: the address (its transport), whether it proved itself there (else sends wait for its proof), and the connection there: libp2p's direct or relayed, Tor's direct (an onion circuit); none on Mycelium; while relayed after a failed hole punch, punching: punches failed in a row (failed) and when it's tried again (nextInSecs).
destination.paths[].rttMs, .down, .connection	peer_path_rtt_seconds{transport}, peer_path_down{transport}	Every path to it, best first: its round-trip time (smoothed, from PINGs), whether it is down, and the transport's connection there (direct or relayed, where it has connections). The first is the one in use: the peer's round-trip time.
lastSyncSecsAgo	peer_last_sync_seconds	Since a sync message from it was applied (or the epoch changed).
divergedForSecs	peer_diverged_seconds	How long its heads have differed from ours with no sync message from it in between; absent while they match. Long values precede a healing re-handshake (peers.md).
syncThroughput	peer_sync_throughput_bytes_per_second	Sync bytes per second over exchanges of at least 32 KiB, round trip taken off, smoothed. Absent before the first large exchange; comes with its age (secsAgo) since large exchanges are rare. This is the rate the peer delivers at in practice (its apply time included), not the path's capacity.
clock.offsetMs, .withinMs	peer_clock_offset_seconds, peer_clock_offset_within_seconds	How far its clock is ahead of ours (behind if negative), give or take. Each PONG says when the peer answered, by its clock, which lies between our PING going and the PONG coming: within half the round trip of its middle, however unevenly the two ways took. Of its paths and the last 5 minutes, the narrowest bound is kept; a sample it contradicts (a clock set anew) replaces it at once. Absent before a PONG from it.
epoch, sameEpoch, heads	—	What it last advertised; sameEpoch: false means we don't sync with it.
handshakeComplete, hasPendingDelivery	—	Session state, for debugging.

Disk

Per kind of the node's own files (state, document, backup, remake), since start: reads, readBytes, writes (whole files), writtenBytes, appends, appendedBytes, errors. Prometheus: disk_…_total{file}. The document file is the only one written steadily: appendedBytes over time is the write rate it costs the device; a rising writes count there means rewrites because the appended part keeps outgrowing the full save (busy swarm, or a --document-rewrite set low).

The dashboard

In builds with the dashboard cargo feature, GET /admin/dashboard is a page showing /admin/stats, /peers and a map of the swarm, live: dashboard.md, and how it works underneath (events, LINKS) at its end. Its LINKS are the one thing a node sends only to measure: the links every member reports, while someone looks.

What is not measured

• Nothing about other nodes but what they send us: no probes, no remote figures (but the dashboard's LINKS).
• Clock skew between nodes that aren't each other's peers: only peers' clocks are compared (clock, per peer).
• Remake durations and outcomes: the remake reports (GET /admin/remakes) hold them.
• Mycelium's own routing, retransmissions and queueing: the daemon's API has them.