From 29cad923a5e2d7dc3207c8d456440f8c37bfd7b6 Mon Sep 17 00:00:00 2001 From: tv Date: Mon, 2 Nov 2026 12:03:02 +0100 Subject: add documentation --- docs/transports.md | 296 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 docs/transports.md (limited to 'docs/transports.md') diff --git a/docs/transports.md b/docs/transports.md new file mode 100644 index 0000000..8fb81d3 --- /dev/null +++ b/docs/transports.md @@ -0,0 +1,296 @@ +# Transports + +How nodes reach each other. A node is its node ID, from its node key; a transport +only says where it is ([`peers.md`](peers.md), node IDs and addresses). Continuum has three +transports, each a cargo feature (`--features mycelium,libp2p,tor`), any of them at +once. Which a node uses is `---enable`, and in which order +`--transport-order` ([`configuration.md`](configuration.md), Transports); the options below are each +transport's. + +| | Mycelium | libp2p | Tor | +|---|---|---|---| +| What it is | An encrypted overlay network, addressed by keys; a daemon per node (supervised, or one running elsewhere) | Direct connections, TCP (Noise, Yamux) and QUIC | An onion service per node (arti, embedded) | +| Through NAT and firewalls | Yes: the overlay routes around them, through its peers | With help: a forwarded or UPnP-mapped port, a relay, hole punching | Yes: onion services need no open port | +| Addresses | `mycelium:` | `libp2p:/p2p/` | `tor:.onion:` | +| Encryption | End to end (the overlay) | Per connection, between the two nodes, through a relay too (Noise, QUIC's TLS) | End to end (Tor) | +| Round trips | Those of the overlay's routes | The network's: the fastest on a LAN or VPN | A second or more | +| Needs | Overlay peers to connect through (`--mycelium-peers`), or a daemon elsewhere | A listen address on at least one side of each pair, or a relay | Time to bootstrap; arti's onion services are still experimental | + +## Several transports + +A node's addresses, one or more per transport, are `addresses` in `GET /admin/stats`. +A node on several transports reaches a peer on the first one in its order that both +have, until the paths are measured, then on the fastest; a send that fails goes over +the next at once ([`peers.md`](peers.md), node IDs and addresses). A node on several also carries +the traffic between nodes that share none, through the document: it syncs with its +neighbours on each side, and the changes travel through its copy. No node forwards +another's messages: a node that bridges two transports holds the document. + +Within one transport, networks are bridged by that transport: a Mycelium daemon (one +without continuum will do) routes for the overlay, and a libp2p relay (a node with +`--libp2p-relay`, or continuum with `--relay-only`, which holds no document) carries +connections between nodes that can't dial each other. + +## Mycelium + +Each node runs its own Mycelium daemon, supervised: started with the node, and +stopped with it. `--mycelium-peers` names the overlay peers it connects through +(`tls://host:9651`, not continuum peers; public Mycelium nodes are used only if listed +there), and `--mycelium-tcp-listen-port` the port it listens on for overlay peers (a +free one without). Its key is its identity on the overlay, and so its address: +`--mycelium-key-file` keeps it across restarts (a new one each start without), and +`--mycelium-key-create` makes it on the first start. `--mycelium-network-key-file` and +`--mycelium-network-name` join a private Mycelium network instead of the public one. + +`--mycelium-rest-url` uses a daemon running elsewhere, through its REST API, instead +of supervising one; then the options above don't apply: those of a supervised daemon +can't be combined with it ([`configuration.md`](configuration.md), unsetting and conflicting options), and +`--mycelium-peers` is left as it is but unused (that daemon has peers of its own). Seeds are a node's address, +its Mycelium public key or IPv6 address, or a host name with an AAAA record for one. + +### When Mycelium doesn't work + +The supervised daemon's own output goes where the node's does, so its complaints are in +the node's log, between the node's lines. + +* **`Starting the supervised Mycelium daemon`** with **``Running `mycelium` ``**: the + `mycelium` program isn't on the node's `PATH`. Supervising needs Linux (the keys are + handed over in memory, and the daemon dies with the node); elsewhere, run the daemon + yourself and give `--mycelium-rest-url`. +* **`Querying local Mycelium identity (no answer within 30s)`**: the daemon didn't answer + its API: it failed to start (its output says why: a port taken, a bad key file), or + `--mycelium-rest-url` points elsewhere. +* **Nobody reaches the node, nor it anyone** (seeds stay unanswered, `refused` grows in + the traffic, `Mycelium rejected outbound message`): the daemons aren't connected on + the overlay. A supervised daemon connects only to `--mycelium-peers` (by default + `tls://127.0.0.1:9651`, a daemon on the same machine) and to who connects to it + (`--mycelium-tcp-listen-port`, which must then be reachable); it doesn't discover + peers on the LAN. Nodes of a private network need the same name and key, or they + don't see each other. +* **A peer's address changes at every restart** (`addressChanges` in the transports' + figures, [`measurements.md`](measurements.md)): it has no `--mycelium-key-file`, so its daemon gets a new + key each start. The node ID stays, so this costs a WHO round, but seeds given as its + Mycelium address go stale. +* **`Discarded messages Mycelium queued for us before we started`** (info), and a few + `staleHelloReplies`: normal after a restart; Mycelium keeps messages for a while. +* **The daemon exits**: the node stops (`A transport stopped`, error), to be restarted + by its supervisor. + +## libp2p + +libp2p connects directly, over TCP (Noise, Yamux) and QUIC: `--libp2p-listen` says +where to listen (e.g. `/ip4/0.0.0.0/tcp/9661`), and the addresses it listens on go +into the member list as `libp2p:/p2p/`, loopback and link-local +ones left out (unless there are no others). `--libp2p-announce` publishes other +addresses instead, e.g. a public address forwarded to the listen port. A port forwarded +on the router needs no announce: once AutoNAT finds the node dialable at the router's +address (peers tell it where they see it, and dial it back there), that address is +published next to the listen addresses. A node's entry in the member list takes its +transports' addresses in turns (each one's best first), 16 at most, so many libp2p +addresses don't crowd out its other transports'. Connected peers +tell each other where they listen (libp2p's identify), so a node also learns where +peers that dialled it can be reached. A node with no reachable listen address can still +take part: it connects out, and its peers answer over that connection (each pair of +peers needs one side that can be dialled). Two peers connected only through a relay +(below) dial each other directly where they know an address for the other; that fails +behind firewalls, so it's tried again after 30 seconds, then less often, up to every 10 +minutes. A directly connected peer's other addresses (from identify) are dialled too, +for a better path: of several connections to a peer, one is kept 20 s after the last +opened, a direct one before a relayed one, then the one with the shortest round trip +measured since then (so a new ethernet path can win over a WLAN connection hours old). + +### Relays + +A node nobody can dial (behind NAT, or with its port closed) can be reached through a +relay instead: a node of the swarm that others can dial, started with `--libp2p-relay`. +A relay says so in the member list (each of its addresses again, ending in +`/p2p-circuit`), and a node that needs a relay picks up to two of those by itself. It +needs one when it has no address others could dial, or when AutoNAT (connected peers +dialling it back) says it can't be dialled; once AutoNAT says it can, it lets them go. +A slot lost is reserved again at once. A relay that refuses a slot (it has none left, +doesn't allow us, isn't a relay, or another peer answers at its address) isn't picked +again for 10 minutes; one that doesn't grant it otherwise (it doesn't answer, or the +connection closed) isn't for 30 seconds. Give relays a `--libp2p-key-file` (with +`--libp2p-key-create`, made on the first start): without +one, a relay's peer ID changes with every start, and the addresses others have for it, +and for the nodes behind it, are wrong until the member list catches up. +`--libp2p-relays /ip4/203.0.113.7/tcp/9661` (with or without the relay's +`/p2p/`) names relays to pick first, before members: used the same way, only +while the node needs a relay. Any libp2p node that serves circuit relay v2 will do, +but most keep circuits short (2 minutes and 128 KiB by default): enough to punch holes, +not to carry traffic where that fails. A relay configured without its peer ID is +connected to, to find it out, and again when it doesn't grant a slot. + +Relays for the whole swarm can be listed in the document instead, so no node needs +them configured: a user in `--protocol-writers` writes their addresses (as `libp2p:` +addresses, with or without `/p2p/`, at most 32) to `_continuum/relays`: + +```sh +curl -u admin:secret -X PUT localhost:3000/store/_continuum/relays \ + -H 'Content-Type: application/json' \ + -d '["libp2p:/ip6/fd00::10/tcp/9661", "libp2p:/ip6/fd00::11/tcp/9661"]' +``` + +Every member picks among them as among configured relays, after those, and before +members that relay; the list goes into new epochs with the rest of `_continuum`. + +A relay need not be a node: `--relay-only` runs continuum as a libp2p relay alone, for +machines that should relay but hold no document (stable hosts on a VPN, say). It has +no node key, joins no swarm and keeps no files but its libp2p key; nodes don't take it +for one of them (it doesn't speak continuum's protocol), nor for a seed: a node given it +as one says so (`Initial HELLO failed`, "a relay, not a node"); its seed is a node behind +the relay (`/p2p-circuit/p2p/`, or `/p2p/` with the relay in +`--libp2p-relays`). It needs `transports = +["libp2p"]`, `libp2p-listen` and `libp2p-key-file` (a relay's peer ID must stay the +same); options about a swarm or a document are errors with it. With `htpasswd`, its +HTTP API has `/health`, `/admin/stats` (its addresses, ready to list, and its +transport figures) and `/admin/log`; without, it has none. + +```toml +relay-only = true +[libp2p] +enable = true +listen = ["/ip6/::/tcp/9661", "/ip6/::/udp/9661/quic-v1"] +key-file = "/var/lib/continuum/libp2p.key" +``` + +Then list it for the swarm (`_continuum/relays`, above), or configure it on the nodes +that need it (`--libp2p-relays`). A node +publishes its address through a relay (`…/p2p//p2p-circuit/p2p/`) with +the others. Peers that reach +it through the relay then try to connect directly (hole punching, libp2p's DCUtR); where +that fails, the relay carries the traffic. Hole punching needs both to listen, though +the firewall may stay closed (`/ip6/::/udp/0/quic-v1`: QUIC sends from where it +listens). Each offers where the relay saw it and where it listens, so nodes on one LAN +meet there, rather than at their router's public address. A punch that fails is tried +again after a minute, then after 5, then every 30 minutes, while the peers stay +connected through the relay: what failed may have been passing. A relay takes at most 1024 circuits (64 per +peer), each for up to an hour and 1 GiB, and doesn't limit how often they're asked for; a circuit that ends is made again on the next +message. Only relays of the swarm are used, unless the node is told to use public +infrastructure (below). How it goes (AutoNAT's verdict, connections direct or relayed, +relay slots, hole punches, failed dials) is in `/admin/stats` under +`transports.libp2p`, and each peer's connection in `/peers` ([`measurements.md`](measurements.md), +Transports). + +### Address ranges + +With `--libp2p-ignore`, libp2p leaves address ranges out, e.g. `10.244.0.0/16` and +`42::/16` for VPNs it should neither depend on nor take a detour through: addresses +there are neither published (announced ones are, whatever they are) nor told peers +that connect nor offered for hole punching nor dialled, and connections there are +refused, whoever dials. Peers connected to the node forget its addresses there; others +when they next connect. Addresses of peers there aren't kept either (nor shown, nor +passed on), nor those through a relay reached there, as they're never dialled. libp2p still listens there, if told to listen +on every interface: the log (at debug level) says which addresses are left out. A node +listening on every interface (`0.0.0.0`, `::`) has many addresses, and a node +publishes 4 direct ones at most (those the router or AutoNAT found first, then those in +`--libp2p-prefer`), +and 2 through relays: leaving VPN ranges out keeps the ones that matter. A relay reached +only through a left-out range (or one without a route) can't be used then, and isn't +tried. + +Ranges this node has no route to are left out alike, without being named: the +operating system is asked whether it has a route to a peer's address (per /24 or /64, +again every 30 s), and where it has none, such addresses aren't kept or dialled. A +node without IPv6 to the internet thus leaves peers' public IPv6 addresses out; once +it has a route again, they're taken again as they're heard of next. Each range that +loses or gets its route is logged (at debug level). + +With `--libp2p-internet-hosts`, e.g. `one.one.one.one` (names or addresses; names are +resolved every 30 s, so both families are covered), the node also asks whether it has a +route to the internet at all, per IPv4 and IPv6: where it has none, it doesn't publish +its own public addresses of that family, which nobody could get answers from; nor any +public address of its own before the first answer (a few seconds after starting). +Nothing is sent to those hosts; only the routing table is asked. Without it, or for a family +none of them resolves to, this isn't asked, and every address is published. + +With `--libp2p-prefer`, the ranges that matter most come first, best first, e.g. the +LAN's `10.23.0.0/16`: of the addresses a node listens on, those there are published +first, and of the addresses a connected peer tells (identify), they are kept first +(a node keeps 8 per peer). Without it, and after the preferred ones, public addresses +come first (IPv4 outside the private, shared and reserved ranges; IPv6 in global +unicast, `2000::/3`), then those through relays, then all others (container bridges, +other LANs, overlays such as Mycelium's `400::/7`). Nodes on one LAN that are connected only through a relay find each +other's LAN address this way, and dial it. A preferred range all inside an ignored one +is refused (nothing there would be used); one with an ignored range inside it is +taken, with a warning: the ignored part stays left out. + +### Through the router: UPnP + +With `--libp2p-upnp`, a node asks its router to forward its listen ports (UPnP, which +most home routers offer, IPv4), and publishes where: the router's public address, next +to its listen addresses. Nodes elsewhere then dial it directly, with neither a relay +nor hole punching, which matters where the other side's NAT defeats punching (mobile +networks' carrier-grade NAT). It needs a fixed port to be useful across restarts +(`/ip4/0.0.0.0/udp/9661/quic-v1`), and different ones for nodes behind one router. The +log says what the router did (`The router forwards to us`), or that no router answered; +a router behind another NAT maps nothing useful, and the log says so too. The ports are +open on the router for as long as the node runs; `--libp2p-announce` takes precedence. + +### Public infrastructure + +With `--libp2p-public`, a node uses public libp2p infrastructure: it joins the IPFS DHT +(Kademlia, through the IPFS bootstrap nodes at `/dnsaddr/bootstrap.libp2p.io`), where it +finds addresses of peers the member list doesn't have, and public peers that relay; +those are picked after members that relay. Public relays keep circuits short (2 minutes +and 128 KiB by default): enough to punch holes through NATs, not to carry a swarm's +traffic for long. It also makes the node's peer ID and addresses known to the DHT, so +to anyone. Without it, nothing public is used. (For Mycelium, public peers are in +`--mycelium-peers`: none unless given.) + +### On the LAN + +With `--libp2p-mdns`, a node finds libp2p peers on its LAN (multicast DNS, IPv4): it +notes their LAN addresses, and those that speak continuum are seeds while it joins, so +nodes on one LAN find each other without `--seeds`. On a LAN with several swarms, a new +node without a state file joins whichever answers first, unless a swarm key +(`--swarm-key-file`) keeps it out of the others. Seeds for it are such +multiaddrs, with or without the peer ID and with or without the `libp2p:` scheme +(`/ip6/fd00::1/tcp/9661`, `libp2p:/ip6/fd00::1/tcp/9661/p2p/12D3…`); without the peer ID, +the node connects to find out who is there. A peer ID alone (`libp2p:/p2p/12D3…`) is +reached where the node knows it, or else through the relays it knows (`--libp2p-relays`, +above): a seed behind a relay, without spelling out the circuit. It doesn't get through NAT yet, and never uses +public libp2p infrastructure (bootstrap nodes, relays). + +## Tor + +Tor (`--tor-enable`, in builds with the `tor` cargo feature) is for reachability: +every node runs an onion service (with arti, embedded), which other nodes reach from +anywhere, without a forwarded port or a relay. It isn't for anonymity: a node's other +addresses are in the member list next to its onion address. `--tor-state-dir` is where +arti keeps its state and cache, the onion service's key among them, which makes its +address: keep it with the node, like the node key. `--tor-port` is the onion service's +port (9662). A node's address is its onion service, `tor:.onion:`, which is +also what a seed gives (one ending in `/`, as nodes published them before, +still works). Arti bootstraps in the background, which takes a while: the node starts +without waiting, sends over Tor wait for it, and a new onion service may take a minute +or two to be reachable. + +Onion services prove themselves to who connects, but not the other way: at the start of +every connection, both sides prove their node key in a challenge handshake bound to the +onion address dialed ([`transport/challenge.rs`](../src/transport/challenge.rs)), and the dialer says which onion service +is its own. The node dialed believes that once a dial there found the same node key: it +dials back to check if it hasn't dialed there yet, once, and closes that connection +again; a claim that failed that check is refused without dialing for 10 minutes. Nodes +keep one connection per pair. Round trips over Tor take a second or more, so a peer on a +faster transport too is reached there while it works (the paths layer picks by +round-trip time). Arti's onion services are still experimental: expect them to be less +reliable than the other transports. + +## Keys + +Each key file is read when the node starts; a missing one is an error, unless the +option creating it is given (a new key, in a new file only its owner may read): + +| Key | File | Created with | A new one means | +|---|---|---|---| +| Node | `--node-key-file` | `--node-key-create` | A new node | +| Swarm | `--swarm-key-file` | `--swarm-key-create` | Only on the swarm's first node: the others need a copy | +| Mycelium | `--mycelium-key-file` | `--mycelium-key-create` | A new Mycelium address | +| libp2p | `--libp2p-key-file` | `--libp2p-key-create` | A new peer ID | + +Tor's onion service key is in `--tor-state-dir`, made by arti. + +## Figures + +How the transports find and reach peers: [`measurements.md`](measurements.md), Transports. -- cgit v1.3.1