From 25076058f102c8275c8fa2ba76efb33a46207687 Mon Sep 17 00:00:00 2001 From: tv Date: Mon, 2 Nov 2026 12:03:02 +0100 Subject: add documentation --- docs/api.md | 247 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 docs/api.md (limited to 'docs/api.md') diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..5b4dac4 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,247 @@ +# The store API, for applications + +What an application can rely on when it reads and writes through a node's HTTP API. The +endpoints and their answers, one by one, are in [`openapi.yaml`](openapi.yaml) (served by every node at +`/openapi.yaml`, with a Swagger UI at `/swagger-ui`); this page is about what they mean +together. + +## The model + +The store is one document, the same on every node once they have synced, read and +written as JSON. A path names a place in it: `settings/theme` is the key `theme` in the +map `settings`, and within a list a segment is an index (`servers/0/name`; `-1` the last +item). Every node holds all of it and answers from its own copy: there is no leader to +ask, and no request goes to another node. + +```sh +curl -u alice:secret -X PUT localhost:3000/store/settings \ + -H 'Content-Type: application/json' -d '{"theme": "dark", "fontSize": 14}' +curl -u alice:secret localhost:3000/store/settings/theme # "dark" +curl -u alice:secret -H 'Accept: text/plain' localhost:3000/store/settings/theme # dark +curl -u alice:secret -X DELETE localhost:3000/store/settings/fontSize +curl -u alice:secret -X POST localhost:3000/store/queue/-1 -d '"job 7"' # append +curl -u alice:secret -X POST 'localhost:3000/store/stats/visits?increment=1' # count +``` + +* `GET /store/` reads, `PUT` writes (what is there then reads as the value + written; the maps on the way are created), `DELETE` deletes (`?prune=true` also + removes the maps it leaves empty), and `POST` inserts into a list at an index, the + items from there moving up, or with `?increment=n` adds `n` to a counter (Concurrent + writes), or with `?cast=` casts what is there to another type (Types). +* Bodies are JSON (`application/json`, the default), text (`text/plain`, stored as a + string), or a value of a type of its own (Types); answers are JSON, or as `Accept` + asks. Numbers in JSON are stored as integers where they fit, as floats otherwise. +* Every request needs a user from the node's `--htpasswd` (basic authentication). Users + are all alike: each can read and write everything but protocol data (below). +* A key with `/` in it, or an empty key, can be written inside a value at its parent, + but not named in a path: read it through the parent. + +## Types + +Every value is stored as one of Automerge's types: a map, a list, a text, or a scalar: +a string, bytes, a signed or unsigned integer, a float, a boolean, null, a counter or a +timestamp (milliseconds since 1970). Every read says which in `x-continuum-type` +(`map`, `list`, `text`, `string`, `bytes`, `int`, `uint`, `float`, `boolean`, `null`, +`counter`, `timestamp`), and so does `?shallow=true` for each entry. + +Each has a JSON form, so any value reads as JSON: bytes as base64, counters and the +integers as numbers, timestamps as RFC 3339 strings (`2026-11-01T21:50:20.345Z`), or as +`Accept: application/json; timestamp-format=…` says: `rfc3339`, `unix` (seconds, with +the milliseconds as a fraction), `unix-ms` or `http-date` (RFC 9110). JSON alone +doesn't tell them apart: written as JSON, values are maps, lists, strings, signed +integers, floats, booleans and nulls. + +A scalar is also read and written as its own media type, its value in the body: + +| Type | Media type | Body | +|---|---|---| +| string, text | `text/plain` | the characters | +| bytes | `application/octet-stream` | the bytes | +| int | `application/vnd.continuum.int` | decimal | +| uint | `application/vnd.continuum.uint` | decimal | +| float | `application/vnd.continuum.float` | decimal | +| boolean | `application/vnd.continuum.boolean` | `true` or `false` | +| counter | `application/vnd.continuum.counter` | decimal (a `PUT` sets it) | +| timestamp | `application/vnd.continuum.timestamp; format=…` | as the format says (`rfc3339` by default) | + +```sh +curl -u alice:secret -X PUT localhost:3000/store/logo -H 'Content-Type: application/octet-stream' --data-binary @logo.png +curl -u alice:secret -H 'Accept: application/octet-stream' localhost:3000/store/logo > logo.png +curl -u alice:secret -X PUT localhost:3000/store/deploy/at \ + -H 'Content-Type: application/vnd.continuum.timestamp; format=unix' -d 1793569820 +curl -u alice:secret -H 'Accept: application/vnd.continuum.timestamp; format=http-date' \ + localhost:3000/store/deploy/at # Sun, 01 Nov 2026 21:50:20 GMT +``` + +Typed JSON (`application/vnd.continuum.typed+json`, for reads and writes) keeps every +type: it is JSON, but for what JSON would show as another type, which is a single-key +object whose key is a tag: `{"$uint": 5}`, `{"$counter": 5}`, `{"$bytes": ""}`, +`{"$timestamp": "2026-11-01T21:50:20.345Z"}` (in its `timestamp-format`) and +`{"$text": "…"}` (a text); a float is a number, as in plain JSON. A map whose only key +starts with `$` is `{"$map": {…}}`, so nothing else is taken for a tag; any other +single `$` key is `400`. A whole map with typed values in it is written at once this +way, and reads back as written. A text written over a text is changed where it differs +(a diff, as splices), so concurrent edits elsewhere in it stay. + +Types are sticky: plain JSON (or text) written over a value takes its type where it is +that type's JSON form. A string over a timestamp that reads as one (RFC 3339, or as the +body's `Content-Type: application/json; timestamp-format=…` says, where numbers are +read as timestamps too), over bytes that is base64, over a text (then changed where it +differs); an integer over a float, an unsigned integer (if it isn't negative) or a +counter (which it sets). So what a client read as JSON writes back as it was stored, +whole maps included. Typed JSON and a type's own media type say the type themselves, +but for an integer over a float, which stays a float in typed JSON too. + +A write that changes the type of what it writes over (a string over a timestamp that +doesn't read as one, a map over a list) says which in `x-continuum-retyped` +(`deploy/at: timestamp -> string`, comma-separated); with `?retype=false` it is refused +instead (`409`, naming the first). A `PUT` answers with what is there after it, read +back, so its `x-continuum-type` is what the value was stored as. + +`POST ?cast=` casts what is there to a type (as `x-continuum-type` names +it), and answers as a `PUT` does: numbers (integers, floats, counters, timestamps as +milliseconds since 1970) to numbers where they are exactly that, any scalar but bytes +to a string or a text as its media type shows it (a timestamp in `?format=`, +`rfc3339` by default) and back, and bytes to a string or a text and back as UTF-8. +Anything else, or a value that isn't one of the type (`"soon"` to a timestamp, `2.5` +to an integer), is `409`, saying why; maps, lists and nulls aren't cast. + +```sh +curl -u alice:secret -X POST 'localhost:3000/store/deploy/at?cast=timestamp&format=unix' +``` + +`Accept` is taken in its order (quality values aren't weighed): the first media type the +value can be shown as, JSON for `*/*` or without an `Accept`, and `406` (naming those it +can be shown as) if none. `text/plain` shows a string as it is and anything else as +JSON. + +## Consistency + +**Reads are local.** A node answers from its copy, which may lack writes other nodes +made a moment ago: they arrive by sync, as fast as messages get from the writer to +this node (a hop between neighbours, a few in a large swarm; [`measurements.md`](measurements.md), +Propagation), and after a partition once it heals. There is +no read that waits for the rest of the swarm. + +**Writes are local too, and never refused for want of others.** A write is applied to +the node's copy at once (the answer holds the value written) and spreads from there. +Two nodes cut off from each other both go on taking writes; when they meet again, the +writes merge. So a write that was answered `200` is durable on that node (with a +`--document-file`, within `--document-append`, 5 s by default) and reaches the others +eventually, but it isn't confirmed by anyone else when answered. + +**Read your own writes** on the node you wrote to: its later reads include the write. +On another node, they include it once synced. `?heads=true` on a read gives the +document's heads (`x-continuum-heads`, the hashes of its latest changes): two reads with +the same heads read the same document. Every write answers with the heads after it, in +the same header. `?at=` reads the document as it was at those heads, on any node +that has them (`404` on one that doesn't, yet), however it changed since: a client that +wrote through one node and reads its write through another reads with the write's +heads, and retries on `404`. + +**One request is one change.** What a `PUT` writes is applied whole or not at all, also +on the other nodes: a map written in one `PUT` never shows half its keys. There are no +transactions across requests: to change several values together, write their common +parent. + +## Concurrent writes + +Writes made on different nodes before they synced are concurrent. They merge, the same +way on every node: + +* **Different keys** both stay: two nodes writing `settings/theme` and + `settings/fontSize` get both. +* **The same key**: one write wins, the same one everywhere. "Newest" means by + Automerge's operation order (the write made after seeing more changes; between equals, + by the writer's actor ID), not by any clock: of two truly concurrent writes, which wins + is arbitrary, but it is the same on every node. +* **Maps written at the same key** merge into one: their keys are united, each key as + above. +* **A write and a delete** of the same key: the write stays (a delete removes only what + the deleting node had seen). +* **List inserts** both stay, in an order every node agrees on. Writing a list item + (`PUT` at its index) overwrites it: two nodes writing the same item concurrently are + two writes of the same key. +* **Counters** (`POST ?increment=n`, made by the first increment) add up: two + nodes incrementing one counter by 1 concurrently leave it 2 higher. Two nodes making + the same counter concurrently make two, and one wins with its increments. +* **Strings** are values, not texts: concurrent writes to one string don't merge + character by character; one wins. + +A write changes only what differs from what is there: a map written over a map keeps +the keys that stay (and what is inside them), gains those it lacks and loses those the +value doesn't have; a list written over a list does the same item by item, by +position; a value equal to the one there isn't written again. So a `PUT` of `settings` +as a whole, concurrent with a `PUT` of `settings/theme` elsewhere, keeps the theme, +unless the whole write changed the theme too (then one wins) or left it out (then the +theme's write stays, as against a delete). Where the path holds concurrent values, or +one of another kind (a map written over a list), it is written anew. + +`?conflicts=true` on a read answers `{"value": …, "conflicts": [...]}`: every place on +the way where concurrent values were resolved, with all candidates and which was +selected (`mergedMaps` or `newestWins`). A later write at that path ends the conflict. + +## Following changes + +`GET /stream/` (or `/stream` for everything) is newline-delimited JSON: one event +with the current value, then one whenever the value there changes, by a write on this +node or by sync: + +``` +{"key":"settings/theme","value":"dark"} +{"key":"settings/theme","value":"light"} +{"key":"settings/theme","deleted":true} +``` + +An event carries the value as a read would return it then. Changes in quick succession +may come as one event with the latest value: a stream tells the value, not every write. +A change that leaves the value as it was sends nothing. The connection stays open; after +it breaks, reconnect, and the first event is the current value again. + +## While a node isn't ready + +A node serves the store once it holds the swarm's document ([`lifecycle.md`](lifecycle.md)): + +* **Joining** (a new node, or one restarted without its document file): `503` for + reads, writes and streams, but for `GET /store`, the whole store, which answers with + what has arrived so far. `GET /ready` (no authentication) says `200` once it is ready. +* **Rejoining** a newer epoch after a remake it missed: reads answer from the document + before, with `x-continuum-stale` naming that epoch; writes are answered as usual and + go into that document (so reads show them), carried over into the new epoch when the + rejoin ends, unless newer writes touched their paths (Concurrent writes); streams are + `503`. +* **Remakes** ([`remakes.md`](remakes.md)) don't interrupt anything else: a node switches to the new + document between two requests, and writes made meanwhile are carried over (at + their path, or where the remake's rules put them; [`remakes.md`](remakes.md), Writes around a + remake). + +Load balancers can send requests to any ready node: `/ready` is the check. + +## Who wrote what + +Every change records the node that committed it (as its Automerge author) and, for a +write through the API, the user who wrote it (in its message, as a trailer like Git's +`Signed-off-by:`: `User: alice`), with when (by the writer's clock). User names are +each node's own (`--htpasswd`), so who wrote it is that user at that node: `alice` on +two nodes is one person only where their operators make it so, which nothing checks. +`GET /admin/changes` lists them, +newest first, with the paths each touched; `GET /admin/changes/settings` those that +touched what is at `settings`. +Every node holds every change, so user names are known to the whole swarm, and so is +who wrote what, until a remake drops the history ([`remakes.md`](remakes.md)). + +## Protocol data + +`_continuum` holds what the nodes coordinate through: the swarm ID, the member registry, +remakes and votes ([`architecture.md`](architecture.md), The document). It is part of the store and shows +in `GET /store`, but isn't application data: only users in `--protocol-writers` may +write it, for repairs. Keep application data out of it, and leave it out when reading +the whole store if it's in the way. + +## Large stores + +`?shallow=true` on a read answers what is there one level deep (a map's keys with their +types and sizes, short scalars' values) instead of the whole value, to browse a large +store a level at a time. Requests are limited to 2 MiB. Every write keeps its history +until a remake compacts it; a document of many small writes grows faster than its +content ([`remakes.md`](remakes.md)). -- cgit v1.3.1