rules: an expired remake aborted by any member

Under a policy, a remake's decision was taken from its coordinator only.
A node aborting an expired remake whose coordinator is gone
(DELETE /admin/remakes, after abortableAt) was refused by its own
judgement (403), so such a remake could never be decided.

The decision is now also taken from a user who may write it (the
protocol writers), and an abort from any member once the remake expired
a grace period ago (abortableAt, by the coordinator's grace period in
the proposal). The rules have no clock of their own, so that is judged
by the time the change says it was made: the writer's word, so a member
could abort a remake early, as it could ask its coordinator to, but
never commit one. Before then, an abort stays a request the coordinator
applies.

4 files changed, 113 insertions, 3 deletions

diff --git a/docs/access.md b/docs/access.md
index 025f47b..7c576b0 100644
--- a/docs/access.md
+++ b/docs/access.md
@@ -226,8 +226,11 @@ stays as it is, wherever it comes from later (a joining node catching up, say).
 * Epochs and the swarm ID (`_continuum/epochs`, `_continuum/swarm`) by an epoch's first
   change only (the genesis change, a remake's snapshot).
 * A remake's record and its proposal by whichever member proposes, where there are
-  none; its decision by the coordinator its proposal names; an abort asked for by any
-  member. The record of the remake that made an epoch, carried into it whole, by the
+  none; its decision by the coordinator its proposal names, or a user who may write
+  there; an abort by any member once the remake expired a grace period ago, by the
+  time the change says it was made (its writer's word: a member could abort a remake
+  early, never commit one); an abort asked for by any member, which the coordinator
+  applies. The record of the remake that made an epoch, carried into it whole, by the
   coordinator its snapshot names, where there is none. Replacing or removing a record or its proposal, by a user who may write
   there.
 * What holds the members' entries, votes and remake records (`_continuum/members`,
diff --git a/docs/remakes.md b/docs/remakes.md
index b70835e..b8e0fa6 100644
--- a/docs/remakes.md
+++ b/docs/remakes.md
@@ -252,6 +252,9 @@ aborts it itself, under three conditions that keep it from crossing a late commi
   Clocks): whether the remake expired is judged by it, and a node ahead would abort a
   remake its coordinator may still commit. Otherwise `409`, unless `?force=clock`.
 
+With a policy ([`access.md`](access.md)), the other nodes take that abort by the time
+it says it was written: a grace period after the expiry at least.
+
 A member that saw a remake expire before it voted for it says so in its log, once
 (`Didn't vote for a remake`, warn if its clock is off: it then misses remakes others
 still vote on).
diff --git a/src/access/rules.rs b/src/access/rules.rs
index cf2740a..a56e5ae 100644
--- a/src/access/rules.rs
+++ b/src/access/rules.rs
@@ -56,6 +56,8 @@ pub(crate) struct Change<'a> {
     pub(crate) paths: &'a BTreeSet<Path>,
     /// Its operations, where each is and what it does there (`inspect::Inspector::ops`).
     pub(crate) ops: &'a [Op],
+    /// When it was made (ms), as its writer says.
+    pub(crate) time: u64,
 }
 
 /// Why a change is refused.
@@ -197,8 +199,13 @@ pub(crate) fn judge(doc: &Document, epoch: Epoch, change: &Change<'_>) -> Result
                 // Its coordinator's; or the record of the remake that made the epoch,
                 // carried into it whole (its proposal written with it) by the coordinator
                 // its snapshot names (`remake::switch`), where there is none.
+                // Also by one who may write it; and an abort by any member, once the
+                // remake expired a grace period ago (`aborts_expired`).
                 Some("decision") => match coordinator(&at, id) {
-                    Some(coordinator) => coordinator == node,
+                    Some(coordinator) => {
+                        coordinator == node || user_may(path) || aborts_expired(&at, id, change)
+                    }
+                    None if user_may(path) => true,
                     None => {
                         change
                             .paths
@@ -284,6 +291,7 @@ pub(crate) fn judge_held(
             user: user.as_deref(),
             paths: &paths,
             ops: &ops,
+            time: u64::try_from(change.timestamp()).unwrap_or(0),
         },
     );
     Some(match judged {
@@ -432,6 +440,38 @@ fn made_the_epoch(at: &At<'_>, epoch: Epoch, id: &str, node: &str) -> bool {
             == Some(node)
 }
 
+/// Whether `change` aborts remake `id`, undecided at `at`, as any member may: once it
+/// expired a grace period ago (`remake::proposal::abortable_at`, its coordinator's grace
+/// period), by the change's time. That is its writer's word: a member could abort a
+/// remake early, as it could ask its coordinator to; never commit one.
+fn aborts_expired(at: &At<'_>, id: &str, change: &Change<'_>) -> bool {
+    if store::read_path(at, &meta::path(&[meta::REMAKES, id, "decision"]))
+        .ok()
+        .flatten()
+        .is_some()
+    {
+        return false;
+    }
+    let outcome = meta::path(&[meta::REMAKES, id, "decision", "outcome"]);
+    let mut outcomes = change.ops.iter().filter(|op| op.path == outcome).peekable();
+    if outcomes.peek().is_none()
+        || !outcomes.all(|op| matches!(&op.action, Action::PutString(value) if value == "aborted"))
+    {
+        return false;
+    }
+    let Some(proposal) = store::read_path(at, &meta::path(&[meta::REMAKES, id, "proposal"]))
+        .ok()
+        .flatten()
+    else {
+        return false;
+    };
+    let number = |key: &str| proposal.get(key).and_then(serde_json::Value::as_u64);
+    let (Some(settle_until), Some(grace)) = (number("settleUntil"), number("graceMs")) else {
+        return false;
+    };
+    change.time >= settle_until.saturating_add(grace.saturating_mul(2))
+}
+
 /// The coordinator remake `id`'s proposal names, at `at`.
 fn coordinator(at: &At<'_>, id: &str) -> Option<String> {
     let proposal = store::read_path(at, &meta::path(&[meta::REMAKES, id, "proposal"]))
@@ -514,6 +554,7 @@ mod tests {
                 user,
                 paths: &touched,
                 ops: &[],
+                time: 0,
             },
         )
     }
@@ -598,6 +639,7 @@ mod tests {
                     user: None,
                     paths: &touched,
                     ops: &[],
+                    time: 0,
                 },
             )
         };
@@ -685,11 +727,67 @@ mod tests {
                 user: None,
                 paths: &touched,
                 ops: &[],
+                time: 0,
             },
         );
         assert!(earlier.is_ok());
     }
 
+    /// A remake's decision: its coordinator's, one who may write it; an abort, any
+    /// member's, once the remake expired a grace period ago, by the change's time.
+    #[test]
+    fn expired_remakes_aborted_by_any_member() {
+        let mut doc = swarm(true);
+        write(
+            &mut doc,
+            "_continuum/remakes/r4/proposal",
+            json!({"coordinator": "a", "settleUntil": 1_000, "graceMs": 100}),
+        );
+        write(
+            &mut doc,
+            "_continuum/users/carol",
+            registry::entry("a", 2, &[(path("_continuum"), &[Right::Write][..])]),
+        );
+        let decide = |user: Option<&str>, outcome: &str, time: u64| {
+            let ops = vec![
+                Op {
+                    path: path("_continuum/remakes/r4/decision"),
+                    action: Action::MakeMap,
+                },
+                Op {
+                    path: path("_continuum/remakes/r4/decision/outcome"),
+                    action: Action::PutString(outcome.to_owned()),
+                },
+                Op {
+                    path: path("_continuum/remakes/r4/decision/decidedAt"),
+                    action: Action::Other,
+                },
+            ];
+            let deps = doc.get_heads();
+            let touched: BTreeSet<Path> = ops.iter().map(|op| op.path.clone()).collect();
+            judge(
+                &doc,
+                Epoch::GENESIS,
+                &Change {
+                    hash: ChangeHash([9; 32]),
+                    deps: &deps,
+                    seen: &deps,
+                    committer: Some("b"),
+                    user,
+                    paths: &touched,
+                    ops: &ops,
+                    time,
+                },
+            )
+        };
+        // Expired at 1100; abortable by others a grace period later.
+        assert!(refused(decide(None, "aborted", 1_199)));
+        assert!(decide(None, "aborted", 1_200).is_ok());
+        assert!(refused(decide(None, "committed", 5_000)));
+        // One who may write protocol data, any decision, any time.
+        assert!(decide(Some("carol"), "committed", 0).is_ok());
+    }
+
     /// A node that isn't a member may only register itself.
     #[test]
     fn strangers_register_themselves_only() {
@@ -750,6 +848,7 @@ mod tests {
                     user: Some("alice"),
                     paths: &touched,
                     ops: &[],
+                    time: 0,
                 },
             )
         };
@@ -812,6 +911,7 @@ mod tests {
                     user: Some("alice"),
                     paths: &touched,
                     ops: &ops,
+                    time: 0,
                 },
             )
         };
@@ -902,6 +1002,7 @@ mod tests {
                     user: Some("alice"),
                     paths: &touched,
                     ops: &ops,
+                    time: 0,
                 },
             )
         };
@@ -979,6 +1080,7 @@ mod tests {
                     user: Some("bob"),
                     paths: &touched,
                     ops: &ops,
+                    time: 0,
                 },
             )
         };
@@ -1033,6 +1135,7 @@ mod tests {
                     user: None,
                     paths: &touched,
                     ops: &[],
+                    time: 0,
                 },
             )
         };
diff --git a/src/access/screen.rs b/src/access/screen.rs
index 37e3adf..a9450a6 100644
--- a/src/access/screen.rs
+++ b/src/access/screen.rs
@@ -414,6 +414,7 @@ pub(crate) fn apply(
                 user: user.as_deref(),
                 paths: &paths[index],
                 ops: &ops[index],
+                time: u64::try_from(change.timestamp()).unwrap_or(0),
             },
         );
         match judged {