continuum

A replicated Automerge document for fleets of machines that aren't always online, read and written as JSON by path (settings/theme, servers/0/name) at any depth. Every node holds the whole document, takes reads and writes on its own, and syncs with its peers whenever it can reach them. Think Syncthing, but for structured data behind an HTTP API.

There is no leader and no quorum for writes: a node keeps working through a network partition, on a laptop on a train, or as the only one left. Concurrent writes are merged, and conflicts are reported rather than hidden.

A node is its key, not its address. Nodes find and reach each other over any of three transports, one or several at once, each a cargo feature:

Mycelium: an encrypted overlay network addressed by keys, so NAT, changing IP addresses and firewalls don't need configuring.
libp2p: direct connections over TCP and QUIC, with relays (members of the swarm, or a relay-only continuum), hole punching, UPnP and LAN discovery for nodes behind NAT.
Tor: an onion service per node (arti, embedded), reachable from anywhere without a forwarded port. For reachability, not anonymity.

A node on several transports reaches each peer over the fastest path they share, falls back to the next when one fails, and carries the traffic between nodes that share none.

The one thing that takes a quorum is a remake: compacting the history, or another move to a new document. A majority of the members must vote for it, so it waits, and in the end aborts, while too many of them are out of reach; reads and writes go on meanwhile (docs/remakes.md).

What it isn't

Continuum is eventually consistent. Reads are local and may be behind; there are no compare-and-set, transactions, leases, locks or leader election. Don't build coordination on it.

Quick start

# The transports are cargo features: mycelium, libp2p, tor (any, at least one);
# dashboard adds the live page (`/admin/dashboard`).
cargo build --release --features mycelium,dashboard

# An HTTP API user (Argon2 hash; e.g. with the `argon2` command-line tool).
printf '%s' 'secret' | argon2 "$(openssl rand -hex 16)" -id -e | sed 's/^/alice:/' > htpasswd

# The first node starts a new swarm. --node-key-create makes the node's key on its first
# start (a new key is a new node; keep the file).
./target/release/continuum --genesis --htpasswd htpasswd --state-file state.json \
  --node-key-file node.key --node-key-create --mycelium-enable

# Further nodes, on other machines, join through any node of it: one of its addresses
# (`addresses` in `GET /admin/stats`: `mycelium:<public key>`, `libp2p:<multiaddr>`,
# `tor:<onion>.onion:<port>`), or for Mycelium its public key or IPv6 address, or a
# host name with an AAAA record for one.
./target/release/continuum --seeds <address of the first node> --htpasswd htpasswd \
  --state-file state.json --node-key-file node.key --node-key-create --mycelium-enable

A node uses the transports enabled, one at least (--mycelium-enable, --libp2p-enable, --tor-enable). With Mycelium, each node runs its own Mycelium daemon, unless pointed at one with --mycelium-rest-url; --mycelium-peers names Mycelium nodes to connect through. With libp2p, --libp2p-listen says where to listen (/ip4/0.0.0.0/tcp/9661), and seeds are multiaddrs (libp2p:/ip4/192.0.2.7/tcp/9661); with Tor (--tor-enable), --tor-state-dir keeps the onion service's key, and seeds are onion addresses. docs/transports.md has the rest. The HTTP API listens on 127.0.0.1:3000:

curl -u alice:secret -X PUT localhost:3000/store/config/feature \
  -H 'Content-Type: application/json' -d '{"enabled": true}'
curl -u alice:secret localhost:3000/store/config/feature
curl -u alice:secret -N localhost:3000/stream/config      # follow changes
curl -u alice:secret localhost:3000/peers                 # who is out there

Documentation

Running nodes

docs/configuration.md: options, files and imports (continuum --help lists them); docs/example.toml has every option, explained.
docs/transports.md: Mycelium, libp2p and Tor: what each is for, how they combine, relays, and their options.
docs/lifecycle.md: starting, joining, rejoining and stopping a node.
docs/operations.md: adding and removing nodes, keys, backups, upgrades, what to alert on.
docs/remakes.md: compacting history and other moves to a new document.
docs/trust.md: who can do what.
docs/measurements.md: /admin/stats and its Prometheus metrics.
docs/dashboard.md: the dashboard, a live page of a node's view of the swarm.

Writing applications

docs/api.md: the store API for applications: consistency, concurrent writes, streams.
docs/openapi.yaml: the HTTP API (also served at /openapi.yaml, with a Swagger UI at /swagger-ui).

Working on continuum

docs/architecture.md: the map of the code, and where to read on.
docs/development.md: building and testing quickly.
docs/peers.md: the protocol between nodes.
docs/glossary.md: the words used in the code, logs, API and docs.