document file: write as a fresh actor once the file is marked closed

With --keep-actor, a node on a file whose last record is Closed writes on
as the actor the header names, which takes the file holding every change
of that actor. At a clean stop, the node's own workers go on writing
until it exits: its alive stamp in the member list, votes, a
coordinator's commit. One committed after Closed was appended, and sent
by the still-running sync, would be a change of that actor the file
lacks, and the next start would make another change with its number.

swarm::document_file::close now takes the document's write lock, marks
the file at the heads it sees there, and sets a fresh actor before
letting go: what the node writes after is no change of the marked actor.
A record the file worker appends after Closed (such a write, flushed)
leaves the file unmarked, which starts a fresh actor next time.

1 files changed, 15 insertions, 2 deletions

diff --git a/src/swarm/document_file.rs b/src/swarm/document_file.rs
index 4c7843d..c833656 100644
--- a/src/swarm/document_file.rs
+++ b/src/swarm/document_file.rs
@@ -28,6 +28,7 @@ pub(crate) async fn run(state: AppState) {
 /// At a clean stop, after the last `flush`: the file marked as holding everything this
 /// node wrote (`DocumentFile::close`), where it does (ready, not rejoining, up to the
 /// document's heads), so a restart may write on as the same actor (`--keep-actor`).
+/// The node writes as a fresh actor from then on.
 pub(crate) async fn close(state: &AppState) {
     let Some(file) = state.document_file.clone() else {
         return;
@@ -35,8 +36,20 @@ pub(crate) async fn close(state: &AppState) {
     if state.remake.is_rejoining() || !state.is_ready().await {
         return;
     }
-    let heads = state.doc.read().await.doc.get_heads();
-    file.close(&heads);
+    // Under the write lock, and leaving the document with a fresh actor: whatever the
+    // node still writes until it exits (its alive stamp, a vote, a coordinator's commit:
+    // nothing stops them) is no change of the actor the file marks as complete.
+    let doc = Arc::clone(&state.doc);
+    let closed = tokio::task::spawn_blocking(move || {
+        let mut replica = doc.blocking_write();
+        let heads = replica.doc.get_heads();
+        file.close(&heads);
+        replica.doc.set_actor(crate::document::ActorId::random());
+    })
+    .await;
+    if let Err(error) = closed {
+        tracing::error!(error = %error, "Marking the document file as closed failed");
+    }
 }
 
 /// Writes what is due now (`DocumentFile::due`). Also called at shutdown.