From 8f08426ae3a2e5f0ecc5d010bc6469b41e3d7678 Mon Sep 17 00:00:00 2001 From: tv Date: Sat, 14 Nov 2026 00:24:45 +0000 Subject: populate: a command line instead of a spec The program takes populate's arguments on its command line, like other programs: populate [OPTION]... SOURCES [TARGET], with the sources as JSON in the file SOURCES, or on standard input for -. TARGET has the syntax of krops' lib's mkTarget, [USER@][HOST][:PORT][PATH], parsed the same way, and what it leaves out gets filled in when populate runs, as mkTarget would, from LOGNAME and the machine's name. Options set what the syntax can't express, or override its parts: --sudo, --ssh-arg, --user, --host, --port, --path, and --local or --remote for whether the target is the machine populate runs on, which populate otherwise decides like isLocalTarget. The sources carry no version. pkgs.populate passes a target attribute set, as mkTarget returns it, part by part, with --local or --remote as evaluation decides, so nothing changes for krops. A target string goes to the program as it is. The program also refuses target hosts and users starting with -, which ssh would take for options, now that a command line can give them. --- pkgs/populate/docs/design.md | 140 +++++++++++++++++++++++-------------------- 1 file changed, 76 insertions(+), 64 deletions(-) (limited to 'pkgs/populate/docs/design.md') diff --git a/pkgs/populate/docs/design.md b/pkgs/populate/docs/design.md index 639d15b..cf9d356 100644 --- a/pkgs/populate/docs/design.md +++ b/pkgs/populate/docs/design.md @@ -2,27 +2,33 @@ Populate delivers sources to a target directory. It consists of two parts: -- Nix code, which turns populate's arguments into a spec: the sources as - `evalSource` returns them, the target, and the options, as JSON. It - checks the spec during evaluation, so mistakes show early. -- A program, which reads a spec and delivers the sources. It knows - nothing about Nix expressions: whatever produces a spec can drive it. +- Nix code, which turns populate's arguments into a command line: the + sources as `evalSource` returns them, as JSON, the target, and the + options. It checks the arguments during evaluation, so mistakes show + early. +- A program, which reads the sources and delivers them to the target its + command line names. It knows nothing about Nix expressions: whatever + produces sources as JSON can drive it. So there is one program for each version of populate, instead of a script for each set of arguments, and the program can be written in any language. This directory holds the program in Rust. Another implementation, e.g. in -Haskell, reads the same spec, prints the same messages, and passes the same -tests. +Haskell, takes the same command line, prints the same messages, and passes +the same tests. ## Nix side `default.nix` is `pkgs.populate`. Given populate's arguments, `{ backup ? false, force ? false, prune ? false, source, target }`, it checks them with -`checks.nix`, turns them into a spec, and returns a script running the -program with the spec on file descriptor 3, so the commands populate runs -keep standard input. The script puts the tools populate runs in front of -`PATH`, and depends on the store paths in the spec, so building it builds -them. `rustProgram` is the program, by default built by `program.nix`. +`checks.nix`, and returns a script running the program with the sources as +JSON on file descriptor 3, so the commands populate runs keep standard +input. A target string goes to the program as it is. A target attribute +set, as `mkTarget` returns it, goes part by part, as options, with +`--local` or `--remote` as krops' lib's `isLocalTarget` decides during +evaluation, like the other functions of krops. The script puts the tools +populate runs in front of `PATH`, and depends on the store paths in the +sources, so building it builds them. `rustProgram` is the program, by +default built by `program.nix`. `writeDeploy` and the other functions of `pkgs.krops` use `pkgs.populate`, so their arguments stay as they are. @@ -30,73 +36,80 @@ so their arguments stay as they are. ## Invocation ``` -populate SPEC +populate [OPTION]... SOURCES [TARGET] ``` -`SPEC` is the path of a file containing the spec, or `-` for standard -input. Exit status: +`SOURCES` is the path of a file containing the sources, or `-` for +standard input. `TARGET` is `[USER@][HOST][:PORT][PATH]`, parsed like +krops' lib's `mkTarget` parses it: the user up to the first `@`, the host +up to a `:` or a `/`, the port up to a `/`, and the path from there. What +it leaves out gets filled in when the program runs, as `mkTarget` would: +the user from `LOGNAME`, the host with the machine's name, and the path +with `/var/src`. An empty user means ssh's default. + +| Option | Meaning | +|-----------------|---------------------------------------------------------| +| `--backup` | Copy the target directory to a backup first. | +| `--force` | Create the target directory's sentinel if it's missing. | +| `--prune` | Remove what isn't a source from the target directory. | +| `--sudo` | Deliver as root, via sudo on the target. | +| `--ssh-arg ARG` | Pass `ARG` to ssh; repeatable. | +| `--user USER` | The target's user, instead of `TARGET`'s. | +| `--host HOST` | The target's host, e.g. one `TARGET` can't express. | +| `--port PORT` | The port ssh connects to. | +| `--path PATH` | The target directory. | +| `--local` | The target is the machine populate runs on. | +| `--remote` | The target isn't, even if it looks so. | +| `-h`, `--help` | Print the usage and exit. | +| `-V`, `--version` | Print populate's version and exit. | + +Options take their argument as the next argument or after `=`, and may +come before, between, or after `SOURCES` and `TARGET`, up to `--`. Exit +status: - 0: populate delivered all sources. -- 1: a check of the spec failed, or delivering did, with a message of its - own. +- 1: a check of the sources or the target failed, or delivering did, with + a message of its own. - The exit status of the command that failed, if delivering failed with that, like the shell implementation does: e.g. 255 if ssh can't reach the target, or 128 if git fetch fails. -- 2: the arguments are wrong, or the spec can't be read, isn't JSON, is of - another version, or doesn't have the spec's shape. +- 2: the command line is wrong, or the sources can't be read, aren't JSON, + or aren't sources. - 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped. -## Spec, version 1 +## Sources -The spec is a JSON object: - -| Field | Type | Default | Meaning | -|-----------|----------------------|---------|-----------------------------------------------| -| `version` | number | | The version of the spec: 1. | -| `backup` | boolean | false | Copy the sources to backup first. | -| `force` | boolean | false | Create the sentinel if it's missing. | -| `prune` | boolean | false | Remove what isn't a source. | -| `source` | object of sources | | The sources, by name. | -| `target` | object | | Where to deliver them. | - -`target` has these fields: - -| Field | Type | Default | Meaning | -|----------------|------------------|---------|------------------------------------------| -| `user` | string or null | null | The user to log in as. | -| `host` | string | | The host to deliver to. | -| `port` | string or null | null | The port ssh connects to. | -| `path` | string | | The target directory. | -| `sudo` | boolean | false | Deliver as root, via sudo. | -| `extraOptions` | array of strings | [] | More arguments for ssh. | -| `local` | boolean or null | null | Whether the target is local. | - -Each source is an object with the field `type`, the source's type, and a -field named after the type, holding the source's settings: +The sources are a JSON object, of sources by name. Each source is an +object with the field `type`, the source's type, and a field named after +the type, holding the source's settings: ```json -{ "type": "git", "git": { "url": "https://...", "ref": "main", ... } } +{ "nixpkgs": { "type": "git", "git": { "url": "https://...", "ref": "main" } } } ``` The types and their settings are those of `lib/types/populate.nix`, as `evalSource` returns them: with shorthands expanded, e.g. `file = "/etc/x"` as `{ "path": "/etc/x" }`, and without settings that are null. -The source types are `derivation`, `file`, `flake`, `flake-input`, `git`, -`pass`, `passage`, `pipe`, and `symlink`. +Settings with defaults may be left out. The source types are +`derivation`, `file`, `flake`, `flake-input`, `git`, `pass`, `passage`, +`pipe`, and `symlink`. -Fields the spec doesn't define are errors, so a misspelt option doesn't go -unnoticed, and a spec of another version gets refused before anything -else. A new version of the spec adds fields or changes their meaning; the -program reads the versions it knows. +Fields and settings populate doesn't know are errors, so a misspelt option +doesn't go unnoticed. The sources carry no version: this is version 0, and +a later version of populate that reads them differently will mark them. -Store paths in the spec, e.g. of a `file` source that is a derivation, must -exist when the program runs. The script `default.nix` writes depends on them. -Something producing a spec at run time has to build them first. +Store paths in the sources, e.g. of a `file` source that is a derivation, +must exist when the program runs. The script `default.nix` writes depends +on them. Something producing sources at run time has to build them first. ## Checks -The program checks the spec as `checks.nix` does, with the same messages: +The program checks the sources and the target as `checks.nix` does, with +the same messages: +- The target's host isn't empty, and neither it nor its user starts with + `-`, so ssh can't take them for options. `checks.nix` doesn't check + this, since krops' targets come from the configuration. - Source names are portable file names: letters, digits, and `._-`, not starting with `-`, and neither `.`, `..`, nor `.populate`. - The target path is absolute, not the root directory, and contains @@ -134,7 +147,7 @@ units of 1024. ## Safety Populate never changes or removes anything outside the target directory, -whatever the spec and whatever it finds on the target: +whatever the sources and whatever it finds on the target: - The checks above keep source names and the target path inside it. - A symlink at the target path of a source gets replaced, never written @@ -152,13 +165,12 @@ places populate must not touch, check this after every phase. ## Execution -Whether a target is local is what the target's `local` says, or, if -that's null, gets decided when the program runs, like krops' lib's -`isLocalTarget` does: it is, if its user is `LOGNAME`, or null if that's -empty or unset, and its host is `localhost` or the host's name, from -`HOSTNAME`, or, if that's empty or unset, the only line of `/etc/hostname` -that is a label. `default.nix` decides during evaluation, like the other -functions of krops. For a local target, populate runs commands directly. +Whether a target is local is what `--local` or `--remote` says, or else +gets decided when the program runs, like krops' lib's `isLocalTarget` +does: it is, if its user is `LOGNAME`, or null if that's empty or unset, +and its host is `localhost` or the machine's name, from `HOSTNAME`, or, if +that's empty or unset, the only line of `/etc/hostname` that is a label. +For a local target, populate runs commands directly. For a remote target, populate runs commands over ssh, sharing one master connection per target. The commands are fixed POSIX shell snippets, with -- cgit v1.3.1