1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
|
# continuum
A replicated [Automerge](https://automerge.org) document for fleets of machines that
aren't always online, read and written as JSON by path (`settings/theme`,
`servers/0/name`) at any depth. Every node holds the whole document, takes reads and
writes on its own, and syncs with its peers whenever it can reach them. Think
Syncthing, but for structured data behind an HTTP API.
There is no leader and no quorum for writes: a node keeps working through a network
partition, on a laptop on a train, or as the only one left. Concurrent writes are
merged, and conflicts are reported rather than hidden.
A node is its key, not its address. Nodes find and reach each other over any of three
transports, one or several at once, each a cargo feature:
* **[Mycelium](https://github.com/threefoldtech/mycelium)**: an encrypted overlay
network addressed by keys, so NAT, changing IP addresses and firewalls don't need
configuring.
* **[libp2p](https://libp2p.io)**: direct connections over TCP and QUIC, with relays
(members of the swarm, or a relay-only continuum), hole punching, UPnP and LAN
discovery for nodes behind NAT.
* **Tor**: an onion service per node (arti, embedded), reachable from anywhere without
a forwarded port. For reachability, not anonymity.
A node on several transports reaches each peer over the fastest path they share, falls
back to the next when one fails, and carries the traffic between nodes that share
none.
The one thing that takes a quorum is a remake: compacting the history, or another move
to a new document. A majority of the members must vote for it, so it waits, and in the
end aborts, while too many of them are out of reach; reads and writes go on meanwhile
([`docs/remakes.md`](docs/remakes.md)).
## What it isn't
Continuum is eventually consistent. Reads are local and may be behind; there are no
compare-and-set, transactions, leases, locks or leader election. Don't build
coordination on it.
## Quick start
```sh
# The transports are cargo features: mycelium, libp2p, tor (any, at least one);
# dashboard adds the live page (`/admin/dashboard`).
cargo build --release --features mycelium,dashboard
# An HTTP API user (Argon2 hash; e.g. with the `argon2` command-line tool).
printf '%s' 'secret' | argon2 "$(openssl rand -hex 16)" -id -e | sed 's/^/alice:/' > htpasswd
# The first node starts a new swarm. --node-key-create makes the node's key on its first
# start (a new key is a new node; keep the file).
./target/release/continuum --genesis --htpasswd htpasswd --state-file state.json \
--node-key-file node.key --node-key-create --mycelium-enable
# Further nodes, on other machines, join through any node of it: one of its addresses
# (`addresses` in `GET /admin/stats`: `mycelium:<public key>`, `libp2p:<multiaddr>`,
# `tor:<onion>.onion:<port>`), or for Mycelium its public key or IPv6 address, or a
# host name with an AAAA record for one.
./target/release/continuum --seeds <address of the first node> --htpasswd htpasswd \
--state-file state.json --node-key-file node.key --node-key-create --mycelium-enable
```
A node uses the transports enabled, one at least (`--mycelium-enable`,
`--libp2p-enable`, `--tor-enable`). With Mycelium, each node runs its own Mycelium
daemon, unless pointed at one with `--mycelium-rest-url`; `--mycelium-peers` names
Mycelium nodes to connect through. With libp2p, `--libp2p-listen` says where to listen
(`/ip4/0.0.0.0/tcp/9661`), and seeds are multiaddrs
(`libp2p:/ip4/192.0.2.7/tcp/9661`); with Tor (`--tor-enable`), `--tor-state-dir`
keeps the onion service's key, and seeds are onion addresses. [`docs/transports.md`](docs/transports.md)
has the rest. The HTTP API listens on `127.0.0.1:3000`:
```sh
curl -u alice:secret -X PUT localhost:3000/store/config/feature \
-H 'Content-Type: application/json' -d '{"enabled": true}'
curl -u alice:secret localhost:3000/store/config/feature
curl -u alice:secret -N localhost:3000/stream/config # follow changes
curl -u alice:secret localhost:3000/peers # who is out there
```
## Documentation
**Running nodes**
* [`docs/configuration.md`](docs/configuration.md): options, files and imports (`continuum --help` lists them);
[`docs/example.toml`](docs/example.toml) has every option, explained.
* [`docs/transports.md`](docs/transports.md): Mycelium, libp2p and Tor: what each is for, how they combine,
relays, and their options.
* [`docs/lifecycle.md`](docs/lifecycle.md): starting, joining, rejoining and stopping a node.
* [`docs/operations.md`](docs/operations.md): adding and removing nodes, keys, backups, upgrades, what to
alert on.
* [`docs/remakes.md`](docs/remakes.md): compacting history and other moves to a new document.
* [`docs/trust.md`](docs/trust.md): who can do what.
* [`docs/measurements.md`](docs/measurements.md): `/admin/stats` and its Prometheus metrics.
* [`docs/dashboard.md`](docs/dashboard.md): the dashboard, a live page of a node's view of the swarm.
**Writing applications**
* [`docs/api.md`](docs/api.md): the store API for applications: consistency, concurrent writes,
streams.
* [`docs/openapi.yaml`](docs/openapi.yaml): the HTTP API (also served at `/openapi.yaml`, with a Swagger
UI at `/swagger-ui`).
**Working on continuum**
* [`docs/architecture.md`](docs/architecture.md): the map of the code, and where to read on.
* [`docs/development.md`](docs/development.md): building and testing quickly.
* [`docs/peers.md`](docs/peers.md): the protocol between nodes.
* [`docs/glossary.md`](docs/glossary.md): the words used in the code, logs, API and docs.
|
