Transports

How nodes reach each other. A node is its node ID, from its node key; a transport only says where it is (peers.md, node IDs and addresses). Continuum has three transports, each a cargo feature (--features mycelium,libp2p,tor), any of them at once. Which a node uses is --<transport>-enable, and in which order --transport-order (configuration.md, Transports); the options below are each transport's.

	Mycelium	libp2p	Tor
What it is	An encrypted overlay network, addressed by keys; a daemon per node (supervised, or one running elsewhere)	Direct connections, TCP (Noise, Yamux) and QUIC	An onion service per node (arti, embedded)
Through NAT and firewalls	Yes: the overlay routes around them, through its peers	With help: a forwarded or UPnP-mapped port, a relay, hole punching	Yes: onion services need no open port
Addresses	`mycelium:<public key>`	`libp2p:<multiaddr>/p2p/<peer ID>`	`tor:<onion>.onion:<port>`
Encryption	End to end (the overlay)	Per connection, between the two nodes, through a relay too (Noise, QUIC's TLS)	End to end (Tor)
Round trips	Those of the overlay's routes	The network's: the fastest on a LAN or VPN	A second or more
Needs	Overlay peers to connect through (`--mycelium-peers`), or a daemon elsewhere	A listen address on at least one side of each pair, or a relay	Time to bootstrap; arti's onion services are still experimental

Several transports

A node's addresses, one or more per transport, are addresses in GET /admin/stats. A node on several transports reaches a peer on the first one in its order that both have, until the paths are measured, then on the fastest; a send that fails goes over the next at once (peers.md, node IDs and addresses). A node on several also carries the traffic between nodes that share none, through the document: it syncs with its neighbours on each side, and the changes travel through its copy. No node forwards another's messages: a node that bridges two transports holds the document.

Within one transport, networks are bridged by that transport: a Mycelium daemon (one without continuum will do) routes for the overlay, and a libp2p relay (a node with --libp2p-relay, or continuum with --relay-only, which holds no document) carries connections between nodes that can't dial each other.

Mycelium

Each node runs its own Mycelium daemon, supervised: started with the node, and stopped with it. --mycelium-peers names the overlay peers it connects through (tls://host:9651, not continuum peers; public Mycelium nodes are used only if listed there), and --mycelium-tcp-listen-port the port it listens on for overlay peers (a free one without). Its key is its identity on the overlay, and so its address: --mycelium-key-file keeps it across restarts (a new one each start without), and --mycelium-key-create makes it on the first start. --mycelium-network-key-file and --mycelium-network-name join a private Mycelium network instead of the public one.

--mycelium-rest-url uses a daemon running elsewhere, through its REST API, instead of supervising one; then the options above don't apply: those of a supervised daemon can't be combined with it (configuration.md, unsetting and conflicting options), and --mycelium-peers is left as it is but unused (that daemon has peers of its own). Seeds are a node's address, its Mycelium public key or IPv6 address, or a host name with an AAAA record for one.

When Mycelium doesn't work

The supervised daemon's own output goes where the node's does, so its complaints are in the node's log, between the node's lines.

Starting the supervised Mycelium daemon with Running `mycelium` : the mycelium program isn't on the node's PATH. Supervising needs Linux (the keys are handed over in memory, and the daemon dies with the node); elsewhere, run the daemon yourself and give --mycelium-rest-url.
Querying local Mycelium identity (no answer within 30s): the daemon didn't answer its API: it failed to start (its output says why: a port taken, a bad key file), or --mycelium-rest-url points elsewhere.
Nobody reaches the node, nor it anyone (seeds stay unanswered, refused grows in the traffic, Mycelium rejected outbound message): the daemons aren't connected on the overlay. A supervised daemon connects only to --mycelium-peers (by default tls://127.0.0.1:9651, a daemon on the same machine) and to who connects to it (--mycelium-tcp-listen-port, which must then be reachable); it doesn't discover peers on the LAN. Nodes of a private network need the same name and key, or they don't see each other.
A peer's address changes at every restart (addressChanges in the transports' figures, measurements.md): it has no --mycelium-key-file, so its daemon gets a new key each start. The node ID stays, so this costs a WHO round, but seeds given as its Mycelium address go stale.
Discarded messages Mycelium queued for us before we started (info), and a few staleHelloReplies: normal after a restart; Mycelium keeps messages for a while.
The daemon exits: the node stops (A transport stopped, error), to be restarted by its supervisor.

libp2p

libp2p connects directly, over TCP (Noise, Yamux) and QUIC: --libp2p-listen says where to listen (e.g. /ip4/0.0.0.0/tcp/9661), and the addresses it listens on go into the member list as libp2p:<multiaddr>/p2p/<peer ID>, loopback and link-local ones left out (unless there are no others). --libp2p-announce publishes other addresses instead, e.g. a public address forwarded to the listen port. A port forwarded on the router needs no announce: once AutoNAT finds the node dialable at the router's address (peers tell it where they see it, and dial it back there), that address is published next to the listen addresses. A node's entry in the member list takes its transports' addresses in turns (each one's best first), 16 at most, so many libp2p addresses don't crowd out its other transports'. Connected peers tell each other where they listen (libp2p's identify), so a node also learns where peers that dialled it can be reached. A node with no reachable listen address can still take part: it connects out, and its peers answer over that connection (each pair of peers needs one side that can be dialled). Two peers connected only through a relay (below) dial each other directly where they know an address for the other; that fails behind firewalls, so it's tried again after 30 seconds, then less often, up to every 10 minutes. A directly connected peer's other addresses (from identify) are dialled too, for a better path: of several connections to a peer, one is kept 20 s after the last opened, a direct one before a relayed one, then the one with the shortest round trip measured since then (so a new ethernet path can win over a WLAN connection hours old).

Relays

A node nobody can dial (behind NAT, or with its port closed) can be reached through a relay instead: a node of the swarm that others can dial, started with --libp2p-relay. A relay says so in the member list (each of its addresses again, ending in /p2p-circuit), and a node that needs a relay picks up to two of those by itself. It needs one when it has no address others could dial, or when AutoNAT (connected peers dialling it back) says it can't be dialled; once AutoNAT says it can, it lets them go. A slot lost is reserved again at once. A relay that refuses a slot (it has none left, doesn't allow us, isn't a relay, or another peer answers at its address) isn't picked again for 10 minutes; one that doesn't grant it otherwise (it doesn't answer, or the connection closed) isn't for 30 seconds. Give relays a --libp2p-key-file (with --libp2p-key-create, made on the first start): without one, a relay's peer ID changes with every start, and the addresses others have for it, and for the nodes behind it, are wrong until the member list catches up. --libp2p-relays /ip4/203.0.113.7/tcp/9661 (with or without the relay's /p2p/<peer ID>) names relays to pick first, before members: used the same way, only while the node needs a relay. Any libp2p node that serves circuit relay v2 will do, but most keep circuits short (2 minutes and 128 KiB by default): enough to punch holes, not to carry traffic where that fails. A relay configured without its peer ID is connected to, to find it out, and again when it doesn't grant a slot.

Relays for the whole swarm can be listed in the document instead, so no node needs them configured: a user in --protocol-writers writes their addresses (as libp2p: addresses, with or without /p2p/<peer ID>, at most 32) to _continuum/relays:

curl -u admin:secret -X PUT localhost:3000/store/_continuum/relays \
  -H 'Content-Type: application/json' \
  -d '["libp2p:/ip6/fd00::10/tcp/9661", "libp2p:/ip6/fd00::11/tcp/9661"]'

Every member picks among them as among configured relays, after those, and before members that relay; the list goes into new epochs with the rest of _continuum.

A relay need not be a node: --relay-only runs continuum as a libp2p relay alone, for machines that should relay but hold no document (stable hosts on a VPN, say). It has no node key, joins no swarm and keeps no files but its libp2p key; nodes don't take it for one of them (it doesn't speak continuum's protocol), nor for a seed: a node given it as one says so (Initial HELLO failed, "a relay, not a node"); its seed is a node behind the relay (<relay>/p2p-circuit/p2p/<peer ID>, or /p2p/<peer ID> with the relay in --libp2p-relays). It needs transports = ["libp2p"], libp2p-listen and libp2p-key-file (a relay's peer ID must stay the same); options about a swarm or a document are errors with it. With htpasswd, its HTTP API has /health, /admin/stats (its addresses, ready to list, and its transport figures) and /admin/log; without, it has none.

relay-only = true
[libp2p]
enable = true
listen = ["/ip6/::/tcp/9661", "/ip6/::/udp/9661/quic-v1"]
key-file = "/var/lib/continuum/libp2p.key"

Then list it for the swarm (_continuum/relays, above), or configure it on the nodes that need it (--libp2p-relays). A node publishes its address through a relay (…/p2p/<relay>/p2p-circuit/p2p/<peer ID>) with the others. Peers that reach it through the relay then try to connect directly (hole punching, libp2p's DCUtR); where that fails, the relay carries the traffic. Hole punching needs both to listen, though the firewall may stay closed (/ip6/::/udp/0/quic-v1: QUIC sends from where it listens). Each offers where the relay saw it and where it listens, so nodes on one LAN meet there, rather than at their router's public address. A punch that fails is tried again after a minute, then after 5, then every 30 minutes, while the peers stay connected through the relay: what failed may have been passing. A relay takes at most 1024 circuits (64 per peer), each for up to an hour and 1 GiB, and doesn't limit how often they're asked for; a circuit that ends is made again on the next message. Only relays of the swarm are used, unless the node is told to use public infrastructure (below). How it goes (AutoNAT's verdict, connections direct or relayed, relay slots, hole punches, failed dials) is in /admin/stats under transports.libp2p, and each peer's connection in /peers (measurements.md, Transports).

Address ranges

With --libp2p-ignore, libp2p leaves address ranges out, e.g. 10.244.0.0/16 and 42::/16 for VPNs it should neither depend on nor take a detour through: addresses there are neither published (announced ones are, whatever they are) nor told peers that connect nor offered for hole punching nor dialled, and connections there are refused, whoever dials. Peers connected to the node forget its addresses there; others when they next connect. Addresses of peers there aren't kept either (nor shown, nor passed on), nor those through a relay reached there, as they're never dialled. libp2p still listens there, if told to listen on every interface: the log (at debug level) says which addresses are left out. A node listening on every interface (0.0.0.0, ::) has many addresses, and a node publishes 4 direct ones at most (those the router or AutoNAT found first, then those in --libp2p-prefer), and 2 through relays: leaving VPN ranges out keeps the ones that matter. A relay reached only through a left-out range (or one without a route) can't be used then, and isn't tried.

Ranges this node has no route to are left out alike, without being named: the operating system is asked whether it has a route to a peer's address (per /24 or /64, again every 30 s), and where it has none, such addresses aren't kept or dialled. A node without IPv6 to the internet thus leaves peers' public IPv6 addresses out; once it has a route again, they're taken again as they're heard of next. Each range that loses or gets its route is logged (at debug level).

With --libp2p-internet-hosts, e.g. one.one.one.one (names or addresses; names are resolved every 30 s, so both families are covered), the node also asks whether it has a route to the internet at all, per IPv4 and IPv6: where it has none, it doesn't publish its own public addresses of that family, which nobody could get answers from; nor any public address of its own before the first answer (a few seconds after starting). Nothing is sent to those hosts; only the routing table is asked. Without it, or for a family none of them resolves to, this isn't asked, and every address is published.

With --libp2p-prefer, the ranges that matter most come first, best first, e.g. the LAN's 10.23.0.0/16: of the addresses a node listens on, those there are published first, and of the addresses a connected peer tells (identify), they are kept first (a node keeps 8 per peer). Without it, and after the preferred ones, public addresses come first (IPv4 outside the private, shared and reserved ranges; IPv6 in global unicast, 2000::/3), then those through relays, then all others (container bridges, other LANs, overlays such as Mycelium's 400::/7). Nodes on one LAN that are connected only through a relay find each other's LAN address this way, and dial it. A preferred range all inside an ignored one is refused (nothing there would be used); one with an ignored range inside it is taken, with a warning: the ignored part stays left out.

Through the router: UPnP

With --libp2p-upnp, a node asks its router to forward its listen ports (UPnP, which most home routers offer, IPv4), and publishes where: the router's public address, next to its listen addresses. Nodes elsewhere then dial it directly, with neither a relay nor hole punching, which matters where the other side's NAT defeats punching (mobile networks' carrier-grade NAT). It needs a fixed port to be useful across restarts (/ip4/0.0.0.0/udp/9661/quic-v1), and different ones for nodes behind one router. The log says what the router did (The router forwards to us), or that no router answered; a router behind another NAT maps nothing useful, and the log says so too. The ports are open on the router for as long as the node runs; --libp2p-announce takes precedence.

Public infrastructure

With --libp2p-public, a node uses public libp2p infrastructure: it joins the IPFS DHT (Kademlia, through the IPFS bootstrap nodes at /dnsaddr/bootstrap.libp2p.io), where it finds addresses of peers the member list doesn't have, and public peers that relay; those are picked after members that relay. Public relays keep circuits short (2 minutes and 128 KiB by default): enough to punch holes through NATs, not to carry a swarm's traffic for long. It also makes the node's peer ID and addresses known to the DHT, so to anyone. Without it, nothing public is used. (For Mycelium, public peers are in --mycelium-peers: none unless given.)

On the LAN

With --libp2p-mdns, a node finds libp2p peers on its LAN (multicast DNS, IPv4): it notes their LAN addresses, and those that speak continuum are seeds while it joins, so nodes on one LAN find each other without --seeds. On a LAN with several swarms, a new node without a state file joins whichever answers first, unless a swarm key (--swarm-key-file) keeps it out of the others. Seeds for it are such multiaddrs, with or without the peer ID and with or without the libp2p: scheme (/ip6/fd00::1/tcp/9661, libp2p:/ip6/fd00::1/tcp/9661/p2p/12D3…); without the peer ID, the node connects to find out who is there. A peer ID alone (libp2p:/p2p/12D3…) is reached where the node knows it, or else through the relays it knows (--libp2p-relays, above): a seed behind a relay, without spelling out the circuit. It doesn't get through NAT yet, and never uses public libp2p infrastructure (bootstrap nodes, relays).

Tor

Tor (--tor-enable, in builds with the tor cargo feature) is for reachability: every node runs an onion service (with arti, embedded), which other nodes reach from anywhere, without a forwarded port or a relay. It isn't for anonymity: a node's other addresses are in the member list next to its onion address. --tor-state-dir is where arti keeps its state and cache, the onion service's key among them, which makes its address: keep it with the node, like the node key. --tor-port is the onion service's port (9662). A node's address is its onion service, tor:<onion>.onion:<port>, which is also what a seed gives (one ending in /<node key>, as nodes published them before, still works). Arti bootstraps in the background, which takes a while: the node starts without waiting, sends over Tor wait for it, and a new onion service may take a minute or two to be reachable.

Onion services prove themselves to who connects, but not the other way: at the start of every connection, both sides prove their node key in a challenge handshake bound to the onion address dialed (transport/challenge.rs), and the dialer says which onion service is its own. The node dialed believes that once a dial there found the same node key: it dials back to check if it hasn't dialed there yet, once, and closes that connection again; a claim that failed that check is refused without dialing for 10 minutes. Nodes keep one connection per pair. Round trips over Tor take a second or more, so a peer on a faster transport too is reached there while it works (the paths layer picks by round-trip time). Arti's onion services are still experimental: expect them to be less reliable than the other transports.

Keys

Each key file is read when the node starts; a missing one is an error, unless the option creating it is given (a new key, in a new file only its owner may read):

Key	File	Created with	A new one means
Node	`--node-key-file`	`--node-key-create`	A new node
Swarm	`--swarm-key-file`	`--swarm-key-create`	Only on the swarm's first node: the others need a copy
Mycelium	`--mycelium-key-file`	`--mycelium-key-create`	A new Mycelium address
libp2p	`--libp2p-key-file`	`--libp2p-key-create`	A new peer ID

Tor's onion service key is in --tor-state-dir, made by arti.

Figures

How the transports find and reach peers: measurements.md, Transports.