path: root/docs/transports.md

# Transports

How nodes reach each other. A node is its node ID, from its node key; a transport
only says where it is ([`peers.md`](peers.md), node IDs and addresses). Continuum has three
transports, each a cargo feature (`--features mycelium,libp2p,tor`), any of them at
once. Which a node uses is `--<transport>-enable`, and in which order
`--transport-order` ([`configuration.md`](configuration.md), Transports); the options below are each
transport's.

| | Mycelium | libp2p | Tor |
|---|---|---|---|
| What it is | An encrypted overlay network, addressed by keys; a daemon per node (supervised, or one running elsewhere) | Direct connections, TCP (Noise, Yamux) and QUIC | An onion service per node (arti, embedded) |
| Through NAT and firewalls | Yes: the overlay routes around them, through its peers | With help: a forwarded or UPnP-mapped port, a relay, hole punching | Yes: onion services need no open port |
| Addresses | `mycelium:<public key>` | `libp2p:<multiaddr>/p2p/<peer ID>` | `tor:<onion>.onion:<port>` |
| Encryption | End to end (the overlay) | Per connection, between the two nodes, through a relay too (Noise, QUIC's TLS) | End to end (Tor) |
| Round trips | Those of the overlay's routes | The network's: the fastest on a LAN or VPN | A second or more |
| Needs | Overlay peers to connect through (`--mycelium-peers`), or a daemon elsewhere | A listen address on at least one side of each pair, or a relay | Time to bootstrap; arti's onion services are still experimental |

## Several transports

A node's addresses, one or more per transport, are `addresses` in `GET /admin/stats`.
A node on several transports reaches a peer on the first one in its order that both
have, until the paths are measured, then on the fastest; a send that fails goes over
the next at once ([`peers.md`](peers.md), node IDs and addresses). A node on several also carries
the traffic between nodes that share none, through the document: it syncs with its
neighbours on each side, and the changes travel through its copy. No node forwards
another's messages: a node that bridges two transports holds the document.

Within one transport, networks are bridged by that transport: a Mycelium daemon (one
without continuum will do) routes for the overlay, and a libp2p relay (a node with
`--libp2p-relay`, or continuum with `--relay-only`, which holds no document) carries
connections between nodes that can't dial each other.

## Mycelium

Each node runs its own Mycelium daemon, supervised: started with the node, and
stopped with it. `--mycelium-peers` names the overlay peers it connects through
(`tls://host:9651`, not continuum peers; public Mycelium nodes are used only if listed
there), and `--mycelium-tcp-listen-port` the port it listens on for overlay peers (a
free one without). Its key is its identity on the overlay, and so its address:
`--mycelium-key-file` keeps it across restarts (a new one each start without), and
`--mycelium-key-create` makes it on the first start. `--mycelium-network-key-file` and
`--mycelium-network-name` join a private Mycelium network instead of the public one.

`--mycelium-rest-url` uses a daemon running elsewhere, through its REST API, instead
of supervising one; then the options above don't apply: those of a supervised daemon
can't be combined with it ([`configuration.md`](configuration.md), unsetting and conflicting options), and
`--mycelium-peers` is left as it is but unused (that daemon has peers of its own). Seeds are a node's address,
its Mycelium public key or IPv6 address, or a host name with an AAAA record for one.

### When Mycelium doesn't work

The supervised daemon's own output goes where the node's does, so its complaints are in
the node's log, between the node's lines.

* **`Starting the supervised Mycelium daemon`** with **``Running `mycelium` ``**: the
  `mycelium` program isn't on the node's `PATH`. Supervising needs Linux (the keys are
  handed over in memory, and the daemon dies with the node); elsewhere, run the daemon
  yourself and give `--mycelium-rest-url`.
* **`Querying local Mycelium identity (no answer within 30s)`**: the daemon didn't answer
  its API: it failed to start (its output says why: a port taken, a bad key file), or
  `--mycelium-rest-url` points elsewhere.
* **Nobody reaches the node, nor it anyone** (seeds stay unanswered, `refused` grows in
  the traffic, `Mycelium rejected outbound message`): the daemons aren't connected on
  the overlay. A supervised daemon connects only to `--mycelium-peers` (by default
  `tls://127.0.0.1:9651`, a daemon on the same machine) and to who connects to it
  (`--mycelium-tcp-listen-port`, which must then be reachable); it doesn't discover
  peers on the LAN. Nodes of a private network need the same name and key, or they
  don't see each other.
* **A peer's address changes at every restart** (`addressChanges` in the transports'
  figures, [`measurements.md`](measurements.md)): it has no `--mycelium-key-file`, so its daemon gets a new
  key each start. The node ID stays, so this costs a WHO round, but seeds given as its
  Mycelium address go stale.
* **`Discarded messages Mycelium queued for us before we started`** (info), and a few
  `staleHelloReplies`: normal after a restart; Mycelium keeps messages for a while.
* **The daemon exits**: the node stops (`A transport stopped`, error), to be restarted
  by its supervisor.

## libp2p

libp2p connects directly, over TCP (Noise, Yamux) and QUIC: `--libp2p-listen` says
where to listen (e.g. `/ip4/0.0.0.0/tcp/9661`), and the addresses it listens on go
into the member list as `libp2p:<multiaddr>/p2p/<peer ID>`, loopback and link-local
ones left out (unless there are no others). `--libp2p-announce` publishes other
addresses instead, e.g. a public address forwarded to the listen port. A port forwarded
on the router needs no announce: once AutoNAT finds the node dialable at the router's
address (peers tell it where they see it, and dial it back there), that address is
published next to the listen addresses. A node's entry in the member list takes its
transports' addresses in turns (each one's best first), 16 at most, so many libp2p
addresses don't crowd out its other transports'. Connected peers
tell each other where they listen (libp2p's identify), so a node also learns where
peers that dialled it can be reached. A node with no reachable listen address can still
take part: it connects out, and its peers answer over that connection (each pair of
peers needs one side that can be dialled). Two peers connected only through a relay
(below) dial each other directly where they know an address for the other; that fails
behind firewalls, so it's tried again after 30 seconds, then less often, up to every 10
minutes. A directly connected peer's other addresses (from identify) are dialled too,
for a better path: of several connections to a peer, one is kept 20 s after the last
opened, a direct one before a relayed one, then the one with the shortest round trip
measured since then (so a new ethernet path can win over a WLAN connection hours old).

### Relays

A node nobody can dial (behind NAT, or with its port closed) can be reached through a
relay instead: a node of the swarm that others can dial, started with `--libp2p-relay`.
A relay says so in the member list (each of its addresses again, ending in
`/p2p-circuit`), and a node that needs a relay picks up to two of those by itself. It
needs one when it has no address others could dial, or when AutoNAT (connected peers
dialling it back) says it can't be dialled; once AutoNAT says it can, it lets them go.
A slot lost is reserved again at once. A relay that refuses a slot (it has none left,
doesn't allow us, isn't a relay, or another peer answers at its address) isn't picked
again for 10 minutes; one that doesn't grant it otherwise (it doesn't answer, or the
connection closed) isn't for 30 seconds. Give relays a `--libp2p-key-file` (with
`--libp2p-key-create`, made on the first start): without
one, a relay's peer ID changes with every start, and the addresses others have for it,
and for the nodes behind it, are wrong until the member list catches up.
`--libp2p-relays /ip4/203.0.113.7/tcp/9661` (with or without the relay's
`/p2p/<peer ID>`) names relays to pick first, before members: used the same way, only
while the node needs a relay. Any libp2p node that serves circuit relay v2 will do,
but most keep circuits short (2 minutes and 128 KiB by default): enough to punch holes,
not to carry traffic where that fails. A relay configured without its peer ID is
connected to, to find it out, and again when it doesn't grant a slot.

Relays for the whole swarm can be listed in the document instead, so no node needs
them configured: a user in `--protocol-writers` writes their addresses (as `libp2p:`
addresses, with or without `/p2p/<peer ID>`, at most 32) to `_continuum/relays`:

```sh
curl -u admin:secret -X PUT localhost:3000/store/_continuum/relays \
  -H 'Content-Type: application/json' \
  -d '["libp2p:/ip6/fd00::10/tcp/9661", "libp2p:/ip6/fd00::11/tcp/9661"]'
```

Every member picks among them as among configured relays, after those, and before
members that relay; the list goes into new epochs with the rest of `_continuum`.

A relay need not be a node: `--relay-only` runs continuum as a libp2p relay alone, for
machines that should relay but hold no document (stable hosts on a VPN, say). It has
no node key, joins no swarm and keeps no files but its libp2p key; nodes don't take it
for one of them (it doesn't speak continuum's protocol), nor for a seed: a node given it
as one says so (`Initial HELLO failed`, "a relay, not a node"); its seed is a node behind
the relay (`<relay>/p2p-circuit/p2p/<peer ID>`, or `/p2p/<peer ID>` with the relay in
`--libp2p-relays`). It needs `transports =
["libp2p"]`, `libp2p-listen` and `libp2p-key-file` (a relay's peer ID must stay the
same); options about a swarm or a document are errors with it. With `htpasswd`, its
HTTP API has `/health`, `/admin/stats` (its addresses, ready to list, and its
transport figures) and `/admin/log`; without, it has none.

```toml
relay-only = true
[libp2p]
enable = true
listen = ["/ip6/::/tcp/9661", "/ip6/::/udp/9661/quic-v1"]
key-file = "/var/lib/continuum/libp2p.key"
```

Then list it for the swarm (`_continuum/relays`, above), or configure it on the nodes
that need it (`--libp2p-relays`). A node
publishes its address through a relay (`…/p2p/<relay>/p2p-circuit/p2p/<peer ID>`) with
the others. Peers that reach
it through the relay then try to connect directly (hole punching, libp2p's DCUtR); where
that fails, the relay carries the traffic. Hole punching needs both to listen, though
the firewall may stay closed (`/ip6/::/udp/0/quic-v1`: QUIC sends from where it
listens). Each offers where the relay saw it and where it listens, so nodes on one LAN
meet there, rather than at their router's public address. A punch that fails is tried
again after a minute, then after 5, then every 30 minutes, while the peers stay
connected through the relay: what failed may have been passing. A relay takes at most 1024 circuits (64 per
peer), each for up to an hour and 1 GiB, and doesn't limit how often they're asked for; a circuit that ends is made again on the next
message. Only relays of the swarm are used, unless the node is told to use public
infrastructure (below). How it goes (AutoNAT's verdict, connections direct or relayed,
relay slots, hole punches, failed dials) is in `/admin/stats` under
`transports.libp2p`, and each peer's connection in `/peers` ([`measurements.md`](measurements.md),
Transports).

### Address ranges

With `--libp2p-ignore`, libp2p leaves address ranges out, e.g. `10.244.0.0/16` and
`42::/16` for VPNs it should neither depend on nor take a detour through: addresses
there are neither published (announced ones are, whatever they are) nor told peers
that connect nor offered for hole punching nor dialled, and connections there are
refused, whoever dials. Peers connected to the node forget its addresses there; others
when they next connect. Addresses of peers there aren't kept either (nor shown, nor
passed on), nor those through a relay reached there, as they're never dialled. libp2p still listens there, if told to listen
on every interface: the log (at debug level) says which addresses are left out. A node
listening on every interface (`0.0.0.0`, `::`) has many addresses, and a node
publishes 4 direct ones at most (those the router or AutoNAT found first, then those in
`--libp2p-prefer`),
and 2 through relays: leaving VPN ranges out keeps the ones that matter. A relay reached
only through a left-out range (or one without a route) can't be used then, and isn't
tried.

Ranges this node has no route to are left out alike, without being named: the
operating system is asked whether it has a route to a peer's address (per /24 or /64,
again every 30 s), and where it has none, such addresses aren't kept or dialled. A
node without IPv6 to the internet thus leaves peers' public IPv6 addresses out; once
it has a route again, they're taken again as they're heard of next. Each range that
loses or gets its route is logged (at debug level).

With `--libp2p-internet-hosts`, e.g. `one.one.one.one` (names or addresses; names are
resolved every 30 s, so both families are covered), the node also asks whether it has a
route to the internet at all, per IPv4 and IPv6: where it has none, it doesn't publish
its own public addresses of that family, which nobody could get answers from; nor any
public address of its own before the first answer (a few seconds after starting).
Nothing is sent to those hosts; only the routing table is asked. Without it, or for a family
none of them resolves to, this isn't asked, and every address is published.

With `--libp2p-prefer`, the ranges that matter most come first, best first, e.g. the
LAN's `10.23.0.0/16`: of the addresses a node listens on, those there are published
first, and of the addresses a connected peer tells (identify), they are kept first
(a node keeps 8 per peer). Without it, and after the preferred ones, public addresses
come first (IPv4 outside the private, shared and reserved ranges; IPv6 in global
unicast, `2000::/3`), then those through relays, then all others (container bridges,
other LANs, overlays such as Mycelium's `400::/7`). Nodes on one LAN that are connected only through a relay find each
other's LAN address this way, and dial it. A preferred range all inside an ignored one
is refused (nothing there would be used); one with an ignored range inside it is
taken, with a warning: the ignored part stays left out.

### Through the router: UPnP

With `--libp2p-upnp`, a node asks its router to forward its listen ports (UPnP, which
most home routers offer, IPv4), and publishes where: the router's public address, next
to its listen addresses. Nodes elsewhere then dial it directly, with neither a relay
nor hole punching, which matters where the other side's NAT defeats punching (mobile
networks' carrier-grade NAT). It needs a fixed port to be useful across restarts
(`/ip4/0.0.0.0/udp/9661/quic-v1`), and different ones for nodes behind one router. The
log says what the router did (`The router forwards to us`), or that no router answered;
a router behind another NAT maps nothing useful, and the log says so too. The ports are
open on the router for as long as the node runs; `--libp2p-announce` takes precedence.

### Public infrastructure

With `--libp2p-public`, a node uses public libp2p infrastructure: it joins the IPFS DHT
(Kademlia, through the IPFS bootstrap nodes at `/dnsaddr/bootstrap.libp2p.io`), where it
finds addresses of peers the member list doesn't have, and public peers that relay;
those are picked after members that relay. Public relays keep circuits short (2 minutes
and 128 KiB by default): enough to punch holes through NATs, not to carry a swarm's
traffic for long. It also makes the node's peer ID and addresses known to the DHT, so
to anyone. Without it, nothing public is used. (For Mycelium, public peers are in
`--mycelium-peers`: none unless given.)

### On the LAN

With `--libp2p-mdns`, a node finds libp2p peers on its LAN (multicast DNS, IPv4): it
notes their LAN addresses, and those that speak continuum are seeds while it joins, so
nodes on one LAN find each other without `--seeds`. On a LAN with several swarms, a new
node without a state file joins whichever answers first, unless a swarm key
(`--swarm-key-file`) keeps it out of the others. Seeds for it are such
multiaddrs, with or without the peer ID and with or without the `libp2p:` scheme
(`/ip6/fd00::1/tcp/9661`, `libp2p:/ip6/fd00::1/tcp/9661/p2p/12D3…`); without the peer ID,
the node connects to find out who is there. A peer ID alone (`libp2p:/p2p/12D3…`) is
reached where the node knows it, or else through the relays it knows (`--libp2p-relays`,
above): a seed behind a relay, without spelling out the circuit. It doesn't get through NAT yet, and never uses
public libp2p infrastructure (bootstrap nodes, relays).

## Tor

Tor (`--tor-enable`, in builds with the `tor` cargo feature) is for reachability:
every node runs an onion service (with arti, embedded), which other nodes reach from
anywhere, without a forwarded port or a relay. It isn't for anonymity: a node's other
addresses are in the member list next to its onion address. `--tor-state-dir` is where
arti keeps its state and cache, the onion service's key among them, which makes its
address: keep it with the node, like the node key. `--tor-port` is the onion service's
port (9662). A node's address is its onion service, `tor:<onion>.onion:<port>`, which is
also what a seed gives (one ending in `/<node key>`, as nodes published them before,
still works). Arti bootstraps in the background, which takes a while: the node starts
without waiting, sends over Tor wait for it, and a new onion service may take a minute
or two to be reachable.

Onion services prove themselves to who connects, but not the other way: at the start of
every connection, both sides prove their node key in a challenge handshake bound to the
onion address dialed ([`transport/challenge.rs`](../src/transport/challenge.rs)), and the dialer says which onion service
is its own. The node dialed believes that once a dial there found the same node key: it
dials back to check if it hasn't dialed there yet, once, and closes that connection
again; a claim that failed that check is refused without dialing for 10 minutes. Nodes
keep one connection per pair. Round trips over Tor take a second or more, so a peer on a
faster transport too is reached there while it works (the paths layer picks by
round-trip time). Arti's onion services are still experimental: expect them to be less
reliable than the other transports.

## Keys

Each key file is read when the node starts; a missing one is an error, unless the
option creating it is given (a new key, in a new file only its owner may read):

| Key | File | Created with | A new one means |
|---|---|---|---|
| Node | `--node-key-file` | `--node-key-create` | A new node |
| Swarm | `--swarm-key-file` | `--swarm-key-create` | Only on the swarm's first node: the others need a copy |
| Mycelium | `--mycelium-key-file` | `--mycelium-key-create` | A new Mycelium address |
| libp2p | `--libp2p-key-file` | `--libp2p-key-create` | A new peer ID |

Tor's onion service key is in `--tor-state-dir`, made by arti.

## Figures

How the transports find and reach peers: [`measurements.md`](measurements.md), Transports.