aboutsummaryrefslogtreecommitdiffstats
path: root/docs/guides/federation/when-things-go-wrong.md
blob: 8c46e1d7f753d62b1f68147c49a876b9ff2223bc (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
# When things go wrong

What to do when a node reports a version it refused, when a party voted twice, when a
node stopped at a version it doesn't read, and when a node doesn't do what it should. The
rules are the design's ([`federation.md`](../../federation.md),
[Changing the policy](../../federation.md#changing-the-policy);
[`store.md`](../../store.md), [A node's own](../../store.md#a-nodes-own)).

## A node refused a version

Two versions changing one are written only where keys voted twice: a party's `true` and
`false` on one proposal, each shown to some, or its `true` on two. Each node takes the
first it judges and refuses the other, with everything built on it, and keeps so across
restarts (`--store-file`); its log says so (`Refused an event its history takes, by what
this node took before: kept so`), and `GET /events/refused` lists it under `own`:

```sh
curl -s -u tv:secret localhost:3000/events/refused | jq .own
curl -s -u tv:secret localhost:3000/events/<ID> | jq .version   # the version it names
curl -s -u tv:secret localhost:3000/events/counts | jq .version  # the one it took
```

Nodes that took different versions are apart: each refuses what is built on the other's.
Nothing about the node that put the refused version is held against it: the version was
valid at its history.

## Finding who voted twice

The votes show it: a party's `true` and `false` on one proposal, or its `true` on two
proposals of one version, neither dead by all the votes. The parties' clients keep their
votes, each as signed, and the votes that freed them (`--record`); gather the records
and ask the client:

```sh
continuum-party twice tv.record lass.record makefu.record
# lass voted twice on version 1f4c…: `true` on 8d02… and 3b71…
```

That proves the party's keys signed both, not that it meant to: two of its clients, two
of its nodes each voting by itself, a vote by hand beside a node voting for it, or a
stolen key look the same. Keep the records: once a checkpoint left those votes out, they
show nowhere else.

## Choosing a side

Which version a node keeps is its operator's choice, and so is joining the other side.
To take the other side's version, have the node drop what it took that parts it, with
everything built on it, and take back what it refused:

```sh
curl -u tv:secret -X POST -H 'Content-Type: application/json' \
  --data '{"ids": ["<the version event it took>"]}' localhost:3000/admin/events/drop
# {"dropped":["…","…"]}
curl -u tv:secret -X POST -H 'Content-Type: application/json' \
  --data '{"ids": ["<the version event it refused>"]}' localhost:3000/admin/events/allow
```

The node refuses what it dropped from then on, from whoever brings it, and writes no
version naming what it dropped, until it is allowed again. A node drops only what it
took after its checkpoint: a version that moved it to a checkpoint it undoes only by
taking a checkpoint its operator pins (`--store-checkpoint`).

The parties answer keys that voted twice by forking, without the party that voted twice
or with its keys named anew ([`forking-and-following.md`](forking-and-following.md)).
Tell the other parties what your node took and what it refused, and send them your
record.

## A node stopped

A node stops at a version event writing a version that passed on the version it reads,
which its continuum doesn't read: of a later federation, using what a later continuum
takes, or none at all. It takes nothing and refuses nothing from then on, and votes
nothing by itself or by hand; what is put through it, it still writes into the document
(`stopped` when put). Its log says so (`The store stopped at an event this continuum
can't judge`), and `GET /events` names the event, why, and how many it holds back:

```sh
curl -s -u tv:secret localhost:3000/events | jq .stopped
# {"heldBack":12,"id":"9a1e…","why":"version 3c0f… passed, and doesn't read as one…"}
```

Upgrade the node to a continuum that reads it: restarted, it reads its store anew and
takes it. Where none does (two thirds passed a proposal that is no version), have the node
drop that event: it goes on under the version before, refuses the event and any writing
that version, and keeps so (`--store-file`):

```sh
curl -u tv:secret -X POST -H 'Content-Type: application/json' \
  --data '{"ids": ["<the event it stopped at>"]}' localhost:3000/admin/events/drop
# {"dropped":["9a1e…"]}
```

Allowed again (`/admin/events/allow`), it stops there once more. A node that stopped
keeps none of it in its store file: restarted, it stops there again.

## A node doing less than it should

* **It holds no store** (`404` from `GET /events`): it hasn't read its document yet, its
  pinned checkpoint isn't in its document yet (`The pinned checkpoint isn't in the
  document (yet)`), or its document's root is no store's (`The document's root is no
  store's: not reading it`).
* **Nobody syncs with it** (`This node isn't a member of its store`): its party's node
  list doesn't name it, as the store reads now. The party adds it to its list
  (`continuum-party nodes --add`), through any member.
* **An event waits** (`waiting` when put): the node lacks an event it builds on. It
  takes it once that one arrives.
* **It doesn't start**: its store file doesn't read, or an option of a swarm that isn't
  federated is set ([`running-a-node.md`](running-a-node.md),
  [Its options](running-a-node.md#its-options)).
* **A write is refused**: [`writing-and-reading.md`](writing-and-reading.md),
  [Why a write is refused](writing-and-reading.md#why-a-write-is-refused).