From 19e8b4ec12a85a030b1fa411acee6b3d7b4b93a7 Mon Sep 17 00:00:00 2001 From: tv Date: Fri, 13 Nov 2026 14:53:03 +0000 Subject: populate docs: describe the design pkgs/populate/docs/design.md describes populate's parts, the program's invocation, the spec, the checks, what populate prints, its safety guarantees, how it runs commands, how to test changes, and what the program does so far. --- pkgs/populate/docs/design.md | 177 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 pkgs/populate/docs/design.md (limited to 'pkgs/populate/docs') diff --git a/pkgs/populate/docs/design.md b/pkgs/populate/docs/design.md new file mode 100644 index 0000000..a312510 --- /dev/null +++ b/pkgs/populate/docs/design.md @@ -0,0 +1,177 @@ +# Populate's design + +Populate delivers sources to a target directory. It consists of two parts: + +- Nix code, which turns populate's arguments into a spec: the sources as + `evalSource` returns them, the target, and the options, as JSON. It + checks the spec during evaluation, so mistakes show early. +- A program, which reads a spec and delivers the sources. It knows + nothing about Nix expressions: whatever produces a spec can drive it. + +So there is one program for each version of populate, instead of a script +for each set of arguments, and the program can be written in any language. +This directory holds the program in Rust. Another implementation, e.g. in +Haskell, reads the same spec, prints the same messages, and passes the same +tests. + +`shell.nix` is the shell implementation, which generates a script for each +set of arguments. It stays until the program delivers everything it does, +and defines the behaviour the program has to match. + +## Nix side + +`default.nix` is `pkgs.populate`. Given populate's arguments, `{ backup ? +false, force ? false, prune ? false, source, target }`, it returns the +command delivering them. `implementation` selects how: + +- `"shell"`, the default: `shell.nix`. +- `"rust"`: `rust.nix`, which checks the arguments with `checks.nix`, turns + them into a spec, and returns a script running the program with the spec + on its standard input. The script sets `PATH` to the tools populate + runs, and depends on the store paths in the spec, so building it builds + them. `rustProgram` is the program, by default built by `program.nix`. + +`writeDeploy` and the other functions of `pkgs.krops` use `pkgs.populate`, +so their arguments stay as they are. + +## Invocation + +``` +populate SPEC +``` + +`SPEC` is the path of a file containing the spec, or `-` for standard +input. Exit status: + +- 0: populate delivered all sources. +- 1: a check of the spec failed, or delivering did. +- 2: the arguments are wrong, or the spec can't be read, isn't JSON, is of + another version, or doesn't have the spec's shape. +- 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped. + +## Spec, version 1 + +The spec is a JSON object: + +| Field | Type | Default | Meaning | +|-----------|----------------------|---------|-----------------------------------------------| +| `version` | number | | The version of the spec: 1. | +| `backup` | boolean | false | Copy the sources to backup first. | +| `force` | boolean | false | Create the sentinel if it's missing. | +| `prune` | boolean | false | Remove what isn't a source. | +| `source` | object of sources | | The sources, by name. | +| `target` | object | | Where to deliver them. | + +`target` has these fields: + +| Field | Type | Default | Meaning | +|----------------|------------------|---------|------------------------------------------| +| `user` | string or null | null | The user to log in as. | +| `host` | string | | The host to deliver to. | +| `port` | string or null | null | The port ssh connects to. | +| `path` | string | | The target directory. | +| `sudo` | boolean | false | Deliver as root, via sudo. | +| `extraOptions` | array of strings | [] | More arguments for ssh. | + +Each source is an object with the field `type`, the source's type, and a +field named after the type, holding the source's settings: + +```json +{ "type": "git", "git": { "url": "https://...", "ref": "main", ... } } +``` + +The types and their settings are those of `lib/types/populate.nix`, as +`evalSource` returns them: with shorthands expanded, e.g. `file = +"/etc/x"` as `{ "path": "/etc/x" }`, and without settings that are null. +The source types are `derivation`, `file`, `flake`, `flake-input`, `git`, +`pass`, `passage`, `pipe`, and `symlink`. + +Fields the spec doesn't define are errors, so a misspelt option doesn't go +unnoticed, and a spec of another version gets refused before anything +else. A new version of the spec adds fields or changes their meaning; the +program reads the versions it knows. + +Store paths in the spec, e.g. of a `file` source that is a derivation, must +exist when the program runs. The script `rust.nix` writes depends on them. +Something producing a spec at run time has to build them first. + +## Checks + +The program checks the spec as `checks.nix` does, with the same messages: + +- Source names are portable file names: letters, digits, and `._-`, not + starting with `-`, and neither `.`, `..`, nor `.populate`. +- The target path is absolute, not the root directory, and contains + neither `..` nor control characters. +- Inputs get drawn from sources that exist, not from the source itself, + and not in circles. + +Checks that depend on where populate runs come on top, e.g. refusing to +copy flakes to targets using sudo, which needs to know whether the target +is local. + +## Output + +What populate prints on standard error is part of its interface: people +read it, and the tests check it. Messages are those of `shell.nix`, e.g.: + +``` +populating 'host:/var/src/nixpkgs'... +pruning 'host:/var/src/old'... +replacing symlink 'host:/var/src/x', which points to '/elsewhere'... +replacing the Git work tree 'host:/var/src/x', because ... +``` + +A target path in a message starts with `user@host:` for remote targets, +without `user@` if the user is root or not set, and is a plain path for +the local target. Errors start with `error:` or `krops:`, warnings with +`warning:`. rsync's itemized changes appear as `%i %n%L %b/%l`. + +## Safety + +Populate never changes or removes anything outside the target directory, +whatever the spec and whatever it finds on the target: + +- The checks above keep source names and the target path inside it. +- A symlink at the target path of a source gets replaced, never written + through, and populate's state directory, `.populate`, must not be one. +- Removing a tree makes only its directories writable, and removes what's + below them, not what symlinks point to. +- Hard links to files of a source get broken before rsync writes to them. +- A Git work tree gets reused only if its `.git` is a directory of its + own, with no other configuration than `git clone` and populate set, + and Git runs without hooks and file system monitors. +- Temporary files get created with `mktemp`, and moved into place. + +The tests' canaries, files around the target directories and in other +places populate must not touch, check this after every phase. + +## Execution + +Whether a target is local gets decided when the program runs: it is, if +its user is the user running populate and its host is `localhost` or the +host's name. For a local target, populate runs commands directly. + +For a remote target, populate runs commands over ssh, sharing one master +connection per target. The commands are fixed POSIX shell snippets, with +values passed as positional arguments, so nothing gets quoted into them. +The target needs a POSIX shell, coreutils, findutils, rsync, Git for +`git` sources, and Nix for flakes; fresh installations of NixOS lack rsync +and Git. With `sudo`, the snippets run via `sudo`. + +Populate runs rsync, Git, ssh, and Nix as programs, found in `PATH`. + +## Development + +- `cargo test` in this directory runs the program's unit tests. +- `nix-build tests --argstr implementation rust -A smoke` runs the smoke + test, a few scenarios covering each source type and each kind of target, + with the program. +- `nix-build tests --argstr implementation rust` runs all scenarios with + the program. A change has to pass them with both implementations, as + long as there are two. + +## Status + +The program reads the spec, checks it, and fails, saying that delivering +sources isn't implemented yet. -- cgit v1.3.1