diff options
Diffstat (limited to 'pkgs')
| -rw-r--r-- | pkgs/populate/README.md | 620 | ||||
| -rw-r--r-- | pkgs/populate/docs/design.md | 93 | ||||
| -rw-r--r-- | pkgs/populate/src/check.rs | 14 |
3 files changed, 635 insertions, 92 deletions
diff --git a/pkgs/populate/README.md b/pkgs/populate/README.md new file mode 100644 index 0000000..f96ef49 --- /dev/null +++ b/pkgs/populate/README.md @@ -0,0 +1,620 @@ +# populate + +populate fills a target directory, `/var/src` by default, on the machine it +runs on or on another one via ssh, with sources: directories and files, +Git work trees, secrets from [pass](https://www.passwordstore.org/) or +[passage](https://github.com/FiloSottile/passage), the output of commands, +Nix derivations, flakes, and inputs of flakes, which can draw their own +inputs from other sources. Running it again brings the target directory up +to date with the sources. It doesn't change anything outside the target +directory. + +[krops](../../README.md) deploys NixOS systems with it. + +## Building + +```sh +nix-build # in this directory, which builds result/bin/populate +cargo build # for development, with Rust 1.71 or later +``` + +`default.nix` takes nixpkgs' `rustPlatform`, from `<nixpkgs>` by default, +so `callPackage` builds it as well. It needs nixpkgs 23.11 or later, and +fetches the program's dependencies from [crates.io](https://crates.io). + + +## Requirements + +populate runs rsync, ssh, Git, Nix, pass, and passage as the sources need +them, found in `PATH`, as well as a POSIX shell, coreutils, and findutils. +The target needs a POSIX shell, coreutils, findutils, rsync, `patch` for +sources with `patches`, Git for `git` sources, and Nix, and, if remote, +ssh access as root, or, with `--sudo`, as a user who may use `sudo`. +NixOS has all of them but Git, and rsync as long as +`environment.defaultPackages` contains it. Features that say so need Nix +with flake support (Nix 2.4 or later) on the machine that uses them. + + +## Command line + +``` +populate [OPTION]... SOURCES [TARGET] +``` + +populate delivers the sources in the file `SOURCES`, or standard input if +`SOURCES` is `-`, to `TARGET`, which is `[USER@][HOST][:PORT][PATH]`. The +user ends at the first `@`, the host at a `:` or a `/`, and the port at a +`/`. The user defaults to `$LOGNAME`, the host to the name of the machine +populate runs on, from `$HOSTNAME` or `/etc/hostname`, and the path to +`/var/src`. A target is local if its user is `$LOGNAME`, and its host is +`localhost` or the name of the machine; populate reaches other targets via +ssh, as their user. + +| Option | Meaning | +|-------------------|---------------------------------------------------------| +| `--backup` | Copy the target directory to a backup first. | +| `--force` | Create the target directory's sentinel if it's missing. | +| `--prune` | Remove what isn't a source from the target directory. | +| `--sudo` | Deliver as root, via sudo on the target. | +| `--ssh-arg ARG` | Pass `ARG` to ssh; repeatable. | +| `--user USER` | The target's user; empty for ssh's default. | +| `--host HOST` | The target's host, e.g. one `TARGET` can't express. | +| `--port PORT` | The port ssh connects to. | +| `--path PATH` | The target directory. | +| `--local` | The target is the machine populate runs on. | +| `--remote` | The target isn't, even if it looks so. | +| `-h`, `--help` | Print the usage and exit. | +| `-V`, `--version` | Print populate's version and exit. | + +`--user`, `--host`, `--port`, and `--path` take precedence over `TARGET`'s +parts. Options take their argument as the next argument or after `=`, and +may come before, between, or after `SOURCES` and `TARGET`, up to `--`. + +Without the sentinel, `.populate` in the target directory, populate refuses +to touch the target directory, unless `--force` creates it; see +[The target directory](#the-target-directory). + +Exit status: + +- 0: populate delivered all sources. +- 1: populate refused the sources or the target, or delivering failed, + with a message of its own. +- The exit status of the command that failed, if delivering failed with + that: e.g. 255 if ssh can't reach the target, or 128 if `git fetch` + fails. +- 2: the command line is wrong, or the sources can't be read, aren't JSON, + or aren't sources. +- 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped. + + +## Sources + +The sources are a JSON object, of sources by name. Each source is an +object with the field `type`, the source's type, and a field named after +the type, holding its settings, which [Source types](#source-types) +describes: + +```json +{ + "nixpkgs": { + "type": "git", + "git": { "url": "https://github.com/NixOS/nixpkgs", "ref": "nixos-26.05" } + }, + "config": { "type": "file", "file": { "path": "/home/me/config" } }, + "nixos-config": { + "type": "symlink", + "symlink": { "target": "config/configuration.nix" } + } +} +``` +```sh +populate --force sources.json root@server +``` + +Settings with defaults may be left out. Fields and settings populate +doesn't know are errors, so a misspelt one doesn't go unnoticed. The +sources carry no version: this is version 0, and a later version of +populate that reads them differently will mark them. + +### Sources in Nix + +`lib` is populate's Nix library: the types of sources, which describe how +to write them in Nix, with shorthands, e.g. `config.file = "/home/me/config"`, +and `evalSource`, which turns sources written so, an attribute set or a list +of them, into the sources populate reads. It takes nixpkgs' lib, from +`<nixpkgs/lib>` by default. + +```sh +nix-instantiate --eval --strict --json -E ' + (import ./lib {}).evalSource { + nixpkgs.git = { + url = "https://github.com/NixOS/nixpkgs"; + ref = "nixos-26.05"; + }; + config.file = "/home/me/config"; + nixos-config.symlink = "config/configuration.nix"; + } +' | populate --force - root@server +``` + +Store paths in the sources, e.g. of a `file` source that is a derivation, +must exist when populate runs. `nix-instantiate --eval` doesn't build +them; building a file that contains the sources, e.g. with `pkgs.writeText +"sources.json" (builtins.toJSON sources)`, builds them too. + + +## The target directory + +Populate owns the target directory, `/var/src` by default: it replaces +whatever is at the target path of a source, and with `prune` it removes +everything else. It doesn't change or remove anything outside: + +* Target paths must be absolute, must not be the root directory, and must not + contain `..`. Source names must consist of letters, digits, and the + characters `._-`, not start with `-`, and be neither `.`, `..`, nor + `.populate`. populate refuses anything else, and so does krops' + `pkgs.populate` during evaluation. + +* A symlink at the target path of a source gets replaced by what the source + delivers, not followed, and populate says so. To keep a source elsewhere, + e.g. on another disk or in memory, mount that place at the target path, or + make the target directory itself a symlink. A Git work tree whose `.git` + is a symlink, or a file naming a repository elsewhere, gets replaced by a + `git` source as well. + +* A `git` source only uses a work tree whose `.git` directory contains no + symlinks, no `commondir` file and no linked work trees, and only the + configuration that `git clone` and populate write, and in which each + `.git` of a submodule is a file naming a repository in `.git/modules`. Otherwise Git's + commands could change what's elsewhere, or run commands, so populating + fails, telling what's wrong, unless the source's `reclone` is set. Git + runs no hooks. + +* A file in the target directory that has other hard links, e.g. one in + `/etc/nixos`, gets replaced by a copy of its own before rsync syncs it, so + the other ones keep their mode and time. Populate writes its own files, + and replaces lock files it rewrites, by renaming new files into place, and + only makes directories writable that it removes. + +* Populate keeps its state in the sentinel, `.populate`, which becomes a + directory once there is state to keep. Populating fails if the sentinel, + or a directory populate needs therein, is a symlink. + + +## Source Types + +### `derivation` + +Nix expression to be built at the target machine. + +Supported attributes: + +* `text` - + Nix expression to be built. + + +### `file` + +The file source type transfers local files (and folders) to the target +using [`rsync`](https://rsync.samba.org/). + +Supported attributes: + +* `path` - + absolute path to files that should by transferred. + +* `useChecksum` (optional) - + boolean that controls whether file contents should be checked to decide + whether a file has changed. This is useful when `path` points at files + with mangled timestamps, e.g. the Nix store. + + The default value is `true` if `path` is a derivation, and `false` otherwise. + +* `filters` (optional) + List of filters that should be passed to [`rsync`](https://rsync.samba.org/). + Filters are specified as attribute sets with the attributes `type` and + `pattern`. Supported filter types are `include` and `exclude`. + Checkout the filter rules section in the + [rsync manual](https://download.samba.org/pub/rsync/rsync.html) + for further information. + +* `deleteExcluded` (optional) + boolean that controls whether the excluded directories should be deleted + if they exist on the target. This is passed to the `--delete-excluded` option + of rsync. Defaults to `true`. + +* `insecureLinks` (optional) - + what to do if rsync can't read `.rsync-filter` files that apply to `path`: + the ones therein, and the ones in the directories above it. rsync 3.5.0 + and later open them more strictly than the files they transfer, and ignore + the ones they can't open, e.g. because a symlink on the way belongs to + neither root nor the user deploying, or, with rsync 3.5.0, because a + directory on the way may be entered but not read, like a home directory + with mode 711. By default, with `null`, populating fails with an error + naming the file. With `true`, rsync reads such files anyway + (`--insecure-links`), which also lets it pass such symlinks on the way to + the target path of a local target; with `false`, rsync syncs without them. + Both print a warning. Older versions of rsync aren't affected. + +* `inputs` (optional) - + inputs of the flake in this source to draw from other sources, see + [Flake inputs from sources](#flake-inputs-from-sources). + +* `rewriteLock` (optional) - + boolean that controls whether inputs get drawn from other sources by + rewriting the lock file of the flake in this source, or else through a + generated flake, see [Flake inputs from sources](#flake-inputs-from-sources). + Defaults to `true`. + + +### `flake` + +The flake source type delivers a flake to the target. The flake gets +archived locally using `nix flake archive`, which includes only files tracked +by Git, so use a `file` source for work trees with untracked files. The +flake and its inputs get copied into the target's Nix store using `nix copy`, +and the target path becomes a symlink to the flake's store path, registered +as garbage collector root. For flakes in a subdirectory, e.g. +`git+file:/src/mono?dir=sub`, the symlink points at that subdirectory of the +store path. The inputs get garbage collector roots in +`.populate/flake-inputs/<name>`, which turns the sentinel file into a +directory. + +Supported attributes: + +* `fetchOnTarget` (optional) - + boolean that controls whether the target fetches the flake's inputs itself + instead of receiving them from the deploying machine. The target then uses + the arguments recorded in the lock file, so inputs that refer to local paths + can only be fetched if the target has these paths as well; populating fails + otherwise, naming the input. If the flake reference names a scheme that + doesn't refer to the deploying machine, e.g. `github:`, the target handles + the flake entirely on its own, and the deploying machine doesn't fetch it. + Otherwise the target fetches the flake itself as well if its locked + reference doesn't refer to the deploying machine, e.g. for indirect + references resolving to `github:` flakes, or else the flake gets + transferred using `rsync`, unless the target's Nix store has it already, + and added to the target's Nix store, where it gets the same store path as + on the deploying machine. This requires Nix with flake support (Nix 2.4 or + later) on the target, and works for targets using `sudo`. Defaults to + `false`. + +* `url` - + flake reference, e.g. the absolute path of a directory containing a + `flake.nix`. Its lock file is used as is; populating fails if the lock + file needs to be updated. + +* `inputs` (optional) - + inputs of the flake in this source to draw from other sources, see + [Flake inputs from sources](#flake-inputs-from-sources). + +* `materialize` (optional) - + boolean that controls whether the target path becomes a writable copy + instead of a link into the store. The copy gets made again only if what + it derives from changes: the delivered store path, the patches, or which + inputs get drawn from which sources. Until then, changes made to it on the + target stay. Nix copies it into the store whenever evaluating a flake + changed it, which is noticeable for big flakes like nixpkgs. For a flake + in a subdirectory, only that subdirectory gets copied. Defaults to + `false`. + +* `patches` (optional) - + list of paths of patches on the deploying machine, which get applied in + order with `patch -p1` on the target, in the directory the target path + shows. Populating fails if a patch doesn't apply. Defaults to `[]`. + +* `rewriteLock` (optional) - + boolean that controls whether inputs get drawn from other sources by + rewriting the lock file of the flake in this source, or else through a + generated flake, see [Flake inputs from sources](#flake-inputs-from-sources). + Defaults to `true`. + +Instead of an attribute set, the flake reference can be given as a string. + +The flake commands on the deploying machine enable flakes themselves, so +flakes don't need to be enabled there. Without `fetchOnTarget`, the target +only needs `nix-store`. Because the copied store paths aren't signed, the +target user must be allowed to import them, i.e. be `root` or a trusted user, +and targets using `sudo` aren't supported. + + +### `flake-input` + +The flake-input source type delivers an input of a flake, as recorded in the +flake's lock file, to the target. This makes the lock file of a flake the +single place that pins, e.g., nixpkgs, for flakes as well as for `NIX_PATH`. +It works like the `flake` source type, but delivers the selected input +instead of the flake, so the target path becomes a symlink to the input's +store path. The inputs that the input's own lock file pins, which evaluating +the input needs, get delivered as well, with garbage collector roots in +`.populate/flake-inputs/<name>`. An input without a lock file comes without +its inputs, because locking it would change it. + +Because the target path points at the store path named by the lock file, it +can be used in `NIX_PATH` as well as by flakes evaluated on the target, +without network access. + +Supported attributes: + +* `url` - + flake reference of the flake whose input gets delivered, e.g. the absolute + path of a directory containing a `flake.nix`. Its lock file is used as is; + populating fails if the lock file needs to be updated. + +* `name` - + name of the input to deliver. Inputs of inputs are separated by `/`, + e.g. `stockholm/nixpkgs`, and resolved as the flake's lock file resolves + them, so an input that follows another one names the other one. + +* `requireInputs` (optional) - + boolean that controls whether populating fails if the inputs that the + input's own lock file pins can't be delivered, e.g. because the lock file + pins something unreachable, or lacks an input its `flake.nix` has. With + `false`, populate prints a warning and delivers the input with the inputs + Nix locks when it adds what the lock file lacks, or else without its + inputs. Evaluating the input as a flake then needs network access. + Defaults to `true`. + +* `fetchOnTarget` (optional) - + boolean that controls whether the target fetches the input itself instead + of receiving it from the deploying machine. The target then uses the + arguments recorded in the lock file, so inputs that refer to local paths + can only be fetched if the target has these paths as well; populating fails + otherwise, naming the input. This requires Nix with flake support (Nix 2.4 + or later) on the target, and works for targets using `sudo`. Defaults to + `false`. + +* `inputs` (optional) - + inputs of the flake in this source to draw from other sources, see + [Flake inputs from sources](#flake-inputs-from-sources). + +* `materialize` (optional) - + boolean that controls whether the target path becomes a writable copy + instead of a link into the store. The copy gets made again only if what + it derives from changes: the delivered store path, the patches, or which + inputs get drawn from which sources. Until then, changes made to it on the + target stay. Nix copies it into the store whenever evaluating a flake + changed it, which is noticeable for big flakes like nixpkgs. For a flake + in a subdirectory, only that subdirectory gets copied. Defaults to + `false`. + +* `patches` (optional) - + list of paths of patches on the deploying machine, which get applied in + order with `patch -p1` on the target, in the directory the target path + shows. Populating fails if a patch doesn't apply. Defaults to `[]`. + +* `rewriteLock` (optional) - + boolean that controls whether inputs get drawn from other sources by + rewriting the lock file of the flake in this source, or else through a + generated flake, see [Flake inputs from sources](#flake-inputs-from-sources). + Defaults to `true`. + +Without `fetchOnTarget`, the same requirements as for the `flake` source type +apply. + +Example: + +```nix +{ + nixpkgs.flake-input = { + url = toString ./.; + name = "nixpkgs"; + }; +} +``` + + +### `git` + +Git sources that will be fetched on the target machine. + +Supported attributes: + +* `url` - + URL of the Git repository that should be fetched. + +* `ref` - + Branch / tag / commit that should be fetched. A branch or tag gets fetched + on every run, so the work tree follows it. A commit, given by its hash or a + prefix thereof of at least four hex digits, only gets fetched if the work + tree lacks it, unless `fetchAlways` is set. So a branch or tag whose name + consists of hex digits only counts as a commit. + +* `clean.exclude` - + List of patterns that should be excluded from Git cleaning. + +* `shallow` (optional) + boolean that controls whether only the requested commit ref. should be fetched + instead of the whole history, to save disk space and bandwith. Defaults to `false`. + +* `reclone` (optional) - + boolean that controls whether a work tree populate can't use, see + [The target directory](#the-target-directory), gets replaced by a fresh + clone, instead of failing. Defaults to `false`. + +* `inputs` (optional) - + inputs of the flake in this source to draw from other sources, see + [Flake inputs from sources](#flake-inputs-from-sources). + +* `rewriteLock` (optional) - + boolean that controls whether inputs get drawn from other sources by + rewriting the lock file of the flake in this source, or else through a + generated flake, see [Flake inputs from sources](#flake-inputs-from-sources). + Defaults to `true`. + + +### `pass` + +The pass source type transfers contents from a local +[password store](https://www.passwordstore.org/) to the target machine. + +Supported attributes: + +* `dir` - + absolute path to the password store. + +* `name` - + sub-directory in the password store. + + +### `passage` + +The passage source type decrypts files from a local +[passage store](https://github.com/FiloSottile/passage) +and transfers them to the target using +[`rsync`](https://rsync.samba.org/). + +Supported attributes: + +* `dir` - + Path to the passage store. + For a partial transfer, this may point to a subdirectory. + Example: `~/.passage/store/hosts/MYHOSTNAME` + +* `identities_file` (optional) - + Path to the identities file. + Defaults to `~/.passage/identities`. + +* `age` (optional) - + Path of the age binary. + Defaults to `age` (absolute path gets resolved using `passage`'s search path.) + + +### `pipe` + +Executes a local command, capture its stdout, and send that as a file to the +target machine. + +Supported attributes: + +* `command` - + The (shell) command to run. + +### `symlink` + +Symlink to create at the target, relative to the target directory. +This can be used to reference files in other sources. + +Supported attributes: + +* `target` - + Content of the symlink. This is typically a relative path. + +* `inputs` (optional) - + inputs of the flake in this source to draw from other sources, see + [Flake inputs from sources](#flake-inputs-from-sources). + + +## Flake inputs from sources + +Sources of type `file`, `flake`, `flake-input`, `git`, and `symlink` can +declare which inputs of the flake they contain get drawn from other sources, +in the style of `flake.nix`, where `follows` names a source instead of an +input. This makes flakes on the target use what populate delivers there, +much like `NIX_PATH` does for Nix expressions. + +```nix +{ + nixpkgs.flake-input = { + url = toString ./.; + name = "nixpkgs"; + }; + crate2nix.flake = { + url = "github:nix-community/crate2nix"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + myconf.file = { + path = toString ./.; + inputs = { + crate2nix.follows = "crate2nix"; + stockholm.inputs.nixpkgs.follows = "nixpkgs"; + }; + }; +} +``` + +Populate rewrites the lock file of the flake in each source that draws +inputs from other sources, using `nix flake lock --override-input`, so the +flake in the source itself uses the other sources. `file` and `git` sources +get their lock file rewritten in place. `flake` and `flake-input` sources +get delivered as a copy with the rewritten lock file, which gets added to the +store, unless materialized, so the target path links to that instead of the +delivered store path, as it does for patched ones. A source drawing an input +from a source that draws inputs itself uses that source as rewritten, so in +the example above, myconf's crate2nix uses nixpkgs from the `nixpkgs` source as +well. + +With `rewriteLock = false`, and for `symlink` sources, which point at other +sources, the source stays as it is, and populate generates a flake in +`.populate/flakes/<name>` instead, which re-exports the outputs of the flake +in the source, with those inputs drawn from the other sources. A source +drawing an input from a source with a generated flake uses that flake. +krops' `writeDeploy` uses a source's generated flake automatically. + +Sources in a Git work tree are referred to as `git+file:` (with submodules if +there are any), all other sources as `path:`, with symlinks resolved. Nix +runs Git in a `git+file:` flake, with its configuration and hooks, so +that's only for work trees whose `.git` directory a `git` source would use, +see [The target directory](#the-target-directory), and has no hooks. Lock +files get rewritten or generated on every run, so they match the sources. +Copies of `flake` and `flake-input` sources get made again only if the +delivered store path, the patches, or the sources they draw inputs from +change. Inputs that get drawn from other sources don't get delivered with +a `flake` or `flake-input` source, and neither does what they pin. +Generated flakes and copies get removed once their sources no longer need +them. This requires Nix with flake support (Nix 2.4 or later) on the +target. Populating fails if a source that draws inputs from other sources +contains no `flake.nix`, or if its flake has no such input, before anything +changes. Sources that inputs get drawn from don't need to be flakes, as long +as the inputs drawing from them are declared with `flake = false`. + + +## Output + +What populate prints on standard error is part of its interface: people +read it, and tests check it. For example: + +``` +populating 'host:/var/src/nixpkgs'... +pruning 'host:/var/src/old'... +replacing symlink 'host:/var/src/x', which points to '/elsewhere'... +replacing the Git work tree 'host:/var/src/x', because ... +``` + +A target path in a message starts with `user@host:` for remote targets, +without `user@` if the user is root or not set, and is a plain path for the +local target. Errors start with `error:` or `krops:`, warnings with +`warning:`. rsync's itemized changes appear as `%i %n%L %'''b/%'''l`, with +sizes in units of 1024. + + +## Tests + +`tests/` contains NixOS VM tests: one test per source type, one for +features spanning sources, such as flake inputs from sources, one for edge +cases of flakes, such as unusual lock files, and one for staying inside the +target directory, whatever links it contains. After every run, all tests +check that nothing outside the target directory changed. The tests deliver +combinations of a source type's settings to the machine populate runs on, +to a remote target as `root`, and to a remote target using `sudo`, and check +the result after delivering and after delivering again. They cover every +combination that a code path tells apart; the exhaustive tests add the +remaining ones, e.g. drawing inputs combined with each of rsync's options. +The scenarios whose results depend on the version of rsync also run with +rsync 3.4.1 and 3.5.0, which `tests/rsync.nix` builds, whatever version +nixpkgs has. The tests require KVM and nixpkgs with +`pkgs.testers.runNixOSTest` (23.05 or later), and use `<nixpkgs>` by +default. + +```sh +nix-build tests # all tests, except the exhaustive ones +nix-build tests -A flake # the test for one source type +nix-build tests -A exhaustive # all exhaustive tests +nix-build tests -A exhaustive.git # the remaining combinations of one test +nix-build tests -A rsync-3-5-0 # the scenarios depending on rsync's version +nix-build tests -A smoke # a few scenarios of each kind, in one VM +nix-build tests -A only --argstr only 'lock-hooks-.*' # scenarios by name +cargo test # the unit tests, without VMs +``` + +[`docs/design.md`](docs/design.md) describes how populate works. diff --git a/pkgs/populate/docs/design.md b/pkgs/populate/docs/design.md index 940ab5f..10e6f4b 100644 --- a/pkgs/populate/docs/design.md +++ b/pkgs/populate/docs/design.md @@ -36,74 +36,13 @@ tools populate runs in front of `PATH`, and depends on the store paths in the sources, so building it builds them. krops' `writeDeploy` and its other functions use `pkgs.populate`, so their arguments stay as they are. -## Invocation +## Command line and sources -``` -populate [OPTION]... SOURCES [TARGET] -``` - -`SOURCES` is the path of a file containing the sources, or `-` for -standard input. `TARGET` is `[USER@][HOST][:PORT][PATH]`, parsed like -krops' lib's `mkTarget` parses it: the user up to the first `@`, the host -up to a `:` or a `/`, the port up to a `/`, and the path from there. What -it leaves out gets filled in when the program runs, as `mkTarget` would: -the user from `LOGNAME`, the host with the machine's name, and the path -with `/var/src`. An empty user means ssh's default. - -| Option | Meaning | -|-----------------|---------------------------------------------------------| -| `--backup` | Copy the target directory to a backup first. | -| `--force` | Create the target directory's sentinel if it's missing. | -| `--prune` | Remove what isn't a source from the target directory. | -| `--sudo` | Deliver as root, via sudo on the target. | -| `--ssh-arg ARG` | Pass `ARG` to ssh; repeatable. | -| `--user USER` | The target's user, instead of `TARGET`'s. | -| `--host HOST` | The target's host, e.g. one `TARGET` can't express. | -| `--port PORT` | The port ssh connects to. | -| `--path PATH` | The target directory. | -| `--local` | The target is the machine populate runs on. | -| `--remote` | The target isn't, even if it looks so. | -| `-h`, `--help` | Print the usage and exit. | -| `-V`, `--version` | Print populate's version and exit. | - -Options take their argument as the next argument or after `=`, and may -come before, between, or after `SOURCES` and `TARGET`, up to `--`. Exit -status: - -- 0: populate delivered all sources. -- 1: a check of the sources or the target failed, or delivering did, with - a message of its own. -- The exit status of the command that failed, if delivering failed with - that, like the shell implementation does: e.g. 255 if ssh can't reach - the target, or 128 if git fetch fails. -- 2: the command line is wrong, or the sources can't be read, aren't JSON, - or aren't sources. -- 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped. - -## Sources - -The sources are a JSON object, of sources by name. Each source is an -object with the field `type`, the source's type, and a field named after -the type, holding the source's settings: - -```json -{ "nixpkgs": { "type": "git", "git": { "url": "https://...", "ref": "main" } } } -``` - -The types and their settings are those of `lib/types/populate.nix`, as -`evalSource` returns them: with shorthands expanded, e.g. `file = -"/etc/x"` as `{ "path": "/etc/x" }`, and without settings that are null. -Settings with defaults may be left out. The source types are -`derivation`, `file`, `flake`, `flake-input`, `git`, `pass`, `passage`, -`pipe`, and `symlink`. - -Fields and settings populate doesn't know are errors, so a misspelt option -doesn't go unnoticed. The sources carry no version: this is version 0, and -a later version of populate that reads them differently will mark them. - -Store paths in the sources, e.g. of a `file` source that is a derivation, -must exist when the program runs. The script krops' `pkgs.populate` -writes depends on them. Something producing sources at run time has to build them first. +[`../README.md`](../README.md) describes populate's command line, the +sources, and what populate prints, which are its interface. The program +reads the sources into `spec::Sources`, and parses the target with +`spec::TargetParts`, completing it with `run::Origin`, the user running +populate and the machine's name, when it runs. ## Checks @@ -129,24 +68,6 @@ refusing to copy flakes to remote targets using sudo. krops' `checks.nix` does that during evaluation too, where `pkgs.populate` decides whether the target is local. -## Output - -What populate prints on standard error is part of its interface: people -read it, and the tests check it. For example: - -``` -populating 'host:/var/src/nixpkgs'... -pruning 'host:/var/src/old'... -replacing symlink 'host:/var/src/x', which points to '/elsewhere'... -replacing the Git work tree 'host:/var/src/x', because ... -``` - -A target path in a message starts with `user@host:` for remote targets, -without `user@` if the user is root or not set, and is a plain path for -the local target. Errors start with `error:` or `krops:`, warnings with -`warning:`. rsync's itemized changes appear as `%i %n%L %'''b/%'''l`, with sizes in -units of 1024. - ## Safety Populate never changes or removes anything outside the target directory, @@ -190,6 +111,8 @@ Populate runs rsync, Git, ssh, and Nix as programs, found in `PATH`. - `nix-build tests -A smoke` runs the smoke test, a few scenarios covering each source type and each kind of target. - `nix-build tests` runs all scenarios but the exhaustive ones. +- krops' tests run populate's scenarios that expect refusals, and those of + the smoke test, through krops' `pkgs.populate`, with `tests/builder.nix`. ## Status diff --git a/pkgs/populate/src/check.rs b/pkgs/populate/src/check.rs index ebe5dbf..5b73deb 100644 --- a/pkgs/populate/src/check.rs +++ b/pkgs/populate/src/check.rs @@ -1,14 +1,14 @@ -//! Checks of the sources and the target, the same as checks.nix makes during -//! evaluation, so sources from anywhere else get checked as well. Messages -//! are the same. +//! Checks of the sources and the target, the same as krops' checks.nix +//! makes during evaluation, so sources from anywhere else get checked as +//! well. Messages are the same. use crate::spec::{Source, Spec}; use serde_json::{Map, Value}; use std::collections::BTreeMap; -/// The checks in the order checks.nix makes them, the first failing one's -/// message. The target's host and user, which a command line gives, come -/// first. +/// The checks in the order krops' checks.nix makes them, the first failing +/// one's message. The target's host and user, which a command line gives, +/// come first. pub fn spec(spec: &Spec) -> Result<(), String> { target_host(&spec.target.host, spec.target.user.as_deref())?; target_path(&spec.target.path)?; @@ -267,7 +267,7 @@ pub fn inputs(sources: &BTreeMap<String, Source>) -> Result<(), String> { } // Depth-first search, with the path of sources on the way, which repeats // a source if drawing goes round in circles. The message names the - // whole path, the way into the circle included, like checks.nix. + // whole path, the way into the circle included, like krops' checks.nix. fn visit<'a>( name: &'a str, drawn_from: &'a BTreeMap<&str, Vec<String>>, |