aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* populate: a command line instead of a spectv9 days7-180/+619
| | | | | | | | | | | | | | | | | | | | | The program takes populate's arguments on its command line, like other programs: populate [OPTION]... SOURCES [TARGET], with the sources as JSON in the file SOURCES, or on standard input for -. TARGET has the syntax of krops' lib's mkTarget, [USER@][HOST][:PORT][PATH], parsed the same way, and what it leaves out gets filled in when populate runs, as mkTarget would, from LOGNAME and the machine's name. Options set what the syntax can't express, or override its parts: --sudo, --ssh-arg, --user, --host, --port, --path, and --local or --remote for whether the target is the machine populate runs on, which populate otherwise decides like isLocalTarget. The sources carry no version. pkgs.populate passes a target attribute set, as mkTarget returns it, part by part, with --local or --remote as evaluation decides, so nothing changes for krops. A target string goes to the program as it is. The program also refuses target hosts and users starting with -, which ssh would take for options, now that a command line can give them.
* populate: filter the program's sources by nametv9 days1-6/+7
| | | | | | | | The filter for the program's sources took the paths it gets relative to toString ./., which they needn't start with, e.g. if a symlink leads to the directory, so nothing matched, and the build found no Cargo.lock. It goes by the names of what it leaves out now: the Nix files, the docs, .gitignore, and cargo's target directory.
* CHANGELOG: 1.31.1, as released from krops-1tv9 days1-0/+7
| | | | | What krops 1.31.1 has goes into a section of its own, the same as in krops-1, leaving what's new in krops 2 unreleased.
* tests: run scenarios selected by nametv9 days2-4/+24
| | | | | | | nix-build tests -A only --argstr only REGEX runs the scenarios whose names match the regular expression REGEX in a single VM test, like the smoke test, e.g. --argstr only 'lock-hooks-.*' for lock-hooks on all three kinds of targets.
* tests: a hook in a flake populate lockstv9 days1-0/+42
| | | | | | | | lock-hooks plants a .git in a file source that keeps it, with configuration a git source would use, but a post-index-change hook, which Nix's Git runs when it adds the lock file it writes into a git+file: flake. Drawing inputs refers to that flake as path:, so the hook doesn't run.
* populate: let git sources keep hooks, refuse them only for git+file:tv9 days3-26/+24
| | | | | | | git sources refused work trees with hooks, but populate's Git doesn't run hooks, so git-foreign's work tree with a hook is fine to use, as it expects. Only Nix's Git runs them, in a git+file: flake, so drawing inputs refers to a work tree with hooks as path: instead.
* tests: check what populate mustn't printtv9 days2-2/+12
| | | | | | | | A phase's unexpected lists what populate is expected not to print, whether it succeeds or not. notdir expects populate not to go on locking after it fails to find the URL of a source drawn from, which would lock the current directory, the login directory on a remote target, copying it into the store.
* populate: state what the target sends back for derived copies in codetv9 days1-32/+140
| | | | | | | | DerivedState::parse reads what derived_state prints, and key_text builds the text whose hash is a derived copy's key, the same as before, both with unit tests, instead of decoding fields by position inside Flakes::derive. Fields from the target that don't end in a null byte fail, instead of getting dropped.
* populate: decide whether a target is local like isLocalTargettv9 days2-13/+86
| | | | | | | | For specs whose target's local is null, the program decides whether the target is local like krops' lib's isLocalTarget: an empty LOGNAME or HOSTNAME counts as unset, as with lib.maybeEnv, and /etc/hostname's labels can be of any length, as with lib.types.label. is_local_with, pure, has unit tests.
* populate: exit with the status of the command that failedtv9 days5-9/+42
| | | | | | | | | | | The shell implementation runs under set -e, so it exits with the status of the command that failed, e.g. 255 if ssh can't reach the target, which wrappers retrying on connection failures rely on, or 128 if git fetch fails. The program does the same, instead of exiting with 1, and with 128 plus the signal for a command a signal ended. A phase's expect.status is the exit status it expects populate to fail with; git-status checks git fetch's.
* populate: check the settings of sources in the spectv9 days3-16/+172
| | | | | | | | | | | | The program checks that each source's settings are those lib/types/populate.nix defines for its type, with values of the kinds it defines, for specs from elsewhere than default.nix: a misspelt setting fails, as design.md promises, instead of being ignored, and so does a setting the program would treat as its default because of its kind. This subsumes the check of absolute paths. ln gets a symlink source's target after --, so it can't be taken for an option either.
* populate: fail if find fails looking for .rsync-filter filestv9 days1-10/+11
| | | | | | | | | | Looking for .rsync-filter files that apply to a file source, a failing find counted as finding none, and only happened with rsync 3.5.0 and later. Like in the shell implementation, find runs before each transfer with filter files, and its failure stops populate with find's message, e.g. for a source that's missing, or contains a directory find can't read, which would also have switched off the check of insecureLinks. A signal during find takes effect right after it.
* populate: give sizes in rsync's log in units of 1024tv9 days2-3/+5
| | | | | | rsync gives sizes in units of 1000 for two apostrophes, and in units of 1024 for three, which the shell implementation asks for. rsync-units checks it.
* populate: check paths in the spec, name the way into a cycletv9 days1-4/+63
| | | | | | | | | The program checks that the paths of file, pass, and pipe sources are absolute, as lib/types/populate.nix demands, for specs from elsewhere, so none can be taken for an option, e.g. of rsync. A cycle of sources drawing inputs from each other gets named with the way into it, like checks.nix names it.
* tests: scenarios for rsync's units and pass changestv9 days1-0/+51
| | | | | | rsync-units checks that rsync's log gives sizes in units of 1024, as --out-format asks for. pass-changes checks that a pass source gets decrypted again when its store changes, and only then.
* populate: run pipe commands through a shelltv9 days2-2/+15
| | | | | | | | A pipe source's command is an absolute path, which a shell runs as a script if it isn't a program, e.g. lacks #!. The program ran it directly, which fails for such a command. pipe-script runs a pipe command without #!.
* populate: refer to flakes as git+file: only with a usable .gittv9 days5-6/+63
| | | | | | | | | | | | | Writing the lock file of a git+file: flake, Nix runs Git, with the repository's configuration and hooks, which may run commands. A .git in a source's directory can come from anywhere, e.g. one planted in a materialized flake, or one a file source keeps. So drawing inputs refers to a flake as git+file: only if git_problems finds nothing wrong with its .git directory, the way git sources check their work trees, and else as path:. lock-foreign-git plants a .git in a materialized flake and in a file source that keeps it, whose configuration names an fsmonitor command.
* populate: check the hooks and submodules of Git work treestv9 days5-6/+116
| | | | | | | | | | | | | | | git submodule update uses the .git of a submodule as it finds it in the work tree: a repository of its own, with configuration that runs commands, or a file naming a repository elsewhere. So git_problems also requires each .git below the work tree to be a file naming a repository in .git/modules. And it refuses hooks in .git/hooks, which populate's Git doesn't run, but Nix's does, writing the lock file of a flake in a git source. git-submodule plants a submodule's .git in the work tree: a repository of its own whose configuration names a filter, and a file naming a repository elsewhere. The fixture withsub has gitsrc as its submodule sub.
* populate: keep URLs, refs, and names from being taken for optionstv9 days3-8/+34
| | | | | | | | | | A git source's url and ref are strings, and git clone, git remote, and git fetch took ones starting with - for options, e.g. --upload-pack, which runs a command where the repository is local. They come after -- now, and so does a pass source's name for git log and pass show. git-ref-option gives a ref that git fetch would take for --upload-pack.
* README: requirements, and where populate's program istv9 days2-4/+26
| | | | | | New users learn what the deploying machine and the target need, including nixpkgs 23.11 or later for building populate, and developers where populate's program, its description, and its unit tests are.
* populate: warn about the arguments it doesn't usetv9 days1-42/+57
| | | | | | | | dash, jq, and implementation, which overrides of the shell implementation set, come through ..., because callPackage sets named arguments, so populate can tell they're set, and warns that they're unused. Other arguments populate doesn't know fail, as they do for named arguments.
* populate: drop the shell implementationtv9 days11-1965/+92
| | | | | | | | | | | | | | | Populate is the program in Rust only. pkgs.populate checks its arguments, and returns a script running the program with them as JSON. The arguments it took for the shell implementation, dash, jq, and implementation, stay accepted and unused, so overrides setting them keep working. Copying flakes to remote targets using sudo gets refused during evaluation, as before, by checks.nix, and by the program for specs from elsewhere. The jq test goes with the jq programs; the program's unit tests cover their equivalents.
* populate: build with Rust 1.71, as in nixpkgs 23.11tv9 days3-3/+4
| | | | | | The program needs Rust 1.71, the oldest version its dependencies support, which Cargo.toml states, so clippy points out what newer versions add. The lock file is of version 3, which Cargo reads from version 1.53 on.
* populate: draw flake inputs natively in Rusttv9 days4-609/+1021
| | | | | | | | | | | | | The program decides how to draw inputs and derive copies of flake and flake-input sources itself. The target runs fixed snippets, which send back what it needs to know: where paths resolve to and what's there, the lock files of flakes as nix flake metadata prints them, and what tells whether a derived copy needs to be made again. The program checks that drawn inputs exist in the lock files, computes the keys of derived copies, and writes generated flakes. Derived copies get the same keys as with the shell implementation, so switching implementations doesn't make them again.
* populate: deliver flake and flake-input sources natively in Rusttv9 days9-623/+1032
| | | | | | | | | | | src/flake_input.rs reads what nix flake metadata and nix flake archive print as JSON itself: it selects the flake or input, decides what the target fetches or gets copied, and computes the store paths to root, which are what selected.jq, selfIsRemote.jq, selfLocked.jq, and the Nix expressions of the shell implementation do there. On the target, fixed snippets in sh::flake run Nix, getting values as positional parameters, and send JSON or store paths back. The jq test's cases are unit tests of the Rust equivalents.
* populate: deliver flakes and draw inputs with the Rust programtv9 days6-40/+1522
| | | | | | | | | | | | | The program delivers flake and flake-input sources, draws inputs from sources, and removes state that sources don't need anymore, so it does everything the shell implementation does, and passes all scenarios. src/flake.rs builds the same shell code as shell.nix generates for these, with every value quoted, so both implementations behave alike while that code moves to Rust piece by piece. The ssh scripts in the temporary directory run the ssh that was in PATH when populate started, so one can be ssh in PATH for nix copy. The other one is for nix copy's ssh:// store, without the control socket.
* populate: deliver sources other than flakes with the Rust programtv9 days10-17/+1435
| | | | | | | | | | | | | | | | | | | | | | The program delivers derivation, file, git, pass, passage, pipe, and symlink sources, makes backups, and prunes, as the shell implementation does, with the same messages. Commands on the target are fixed shell snippets in src/sh.rs, which get values as positional parameters, run directly for local targets and over ssh, sharing one master connection, for remote ones. rsync's transfers and its check for .rsync-filter files it can't read are in src/rsync.rs. Given flake or flake-input sources, or sources drawing inputs from others, the program fails before changing anything, saying that these aren't implemented yet. SIGHUP, SIGINT, and SIGTERM make populate stop once the command it waits for ends, stop the ssh master, remove its temporary directory, and exit with 128 plus the signal's number. The spec's target gets local, which rust.nix sets during evaluation, like the other functions of krops decide, and which the program decides when it runs if it's null. The spec comes on file descriptor 3, so the commands populate runs keep standard input, and the tools populate runs come first in PATH.
* populate docs: describe the designtv9 days1-0/+177
| | | | | | | pkgs/populate/docs/design.md describes populate's parts, the program's invocation, the spec, the checks, what populate prints, its safety guarantees, how it runs commands, how to test changes, and what the program does so far.
* populate: add a Rust implementation, selectable for the teststv9 days13-1616/+2247
| | | | | | | | | | | | | | | | | | | pkgs.populate takes implementation: "shell", the default, generates a script from populate's arguments, and "rust" passes them as JSON to the program in pkgs/populate, built from the crate there by program.nix. The program checks the spec as checks.nix does during evaluation, with the same messages, and reports that delivering sources isn't implemented yet. rustProgram replaces the program, e.g. with one built some other way. checks.nix holds the checks of populate's arguments during evaluation. Both implementations check the names of sources and the target path with it, and the Rust one's wrapper also checks drawing inputs, which the shell implementation checks where it draws them. nix-build tests --argstr implementation rust runs the scenarios with the Rust implementation. They expect it to refuse copying flakes to targets using sudo when it runs, not during evaluation, because only then it decides whether the target is local.
* tests: run populate with /dev/null as inputtv9 days1-2/+4
| | | | | | | | | | The test driver runs commands in a shell whose input is the console it talks through, /dev/hvc0, which is a terminal. ssh -t, which writeDeploy's allocateTTY passes for remote targets, allocates a terminal then, and reads the console, so the deploy-*-allocateTTY scenarios of remote and sudo targets hang. Populate and writeDeploy read nothing, so their input is /dev/null, and ssh allocates no terminal, as these scenarios expect.
* tests: add a smoke testtv9 days1-2/+45
| | | | | | | | | | nix-build tests -A smoke runs 19 scenarios in a single VM test: each source type, each kind of target, flakes drawing inputs both ways, the sentinel, prune, the checks of source names and target paths, and writeDeploy. That's quick enough to run after every change, while the complete tests stay what a change has to pass in the end. nix-build tests doesn't build it, because the complete tests run the same scenarios.
* populate, overlay: mark populate's scripts as shell, drop unused libtv9 days2-6/+2
| | | | | | The scripts populate writes with writeDash get /* sh */ markers like the other shell code, and the overlay doesn't import krops' lib, which it doesn't use.
* populate: omit the times of directories and symlinkstv9 days1-0/+4
| | | | | | | rsync sets the modification time of each directory whose contents change, and logs a line for it. Nothing relies on these times, nor on the times of symlinks, so --omit-dir-times and --omit-link-times leave them alone, and the log lists only what got transferred or deleted.
* krops: add allocateTTY to writeDeploytv9 days5-6/+41
| | | | | | | | | With allocateTTY, nixos-rebuild runs with a terminal, so it shows colours and progress: on remote targets, ssh gets -t, which allocates a terminal if its own input is one, and on the local target, expect's unbuffer provides one. The nixos-rebuild stand-in records whether its output is a terminal.
* populate: follow branches and tags of git sourcestv9 days4-18/+69
| | | | | | | | A ref that consists of 4 to 40 hex digits is a commit, which populate fetches only if the work tree lacks it, unless fetchAlways is set. Any other ref is a branch or tag, which populate fetches on every run, resetting the work tree to the commit fetched. Resetting to the name itself resolves to a local branch, which fetching doesn't move.
* CHANGELOG: add 1.31.0v1.31.0tv9 days1-0/+29
|
* populate: name the target alike in all messagestv9 days2-29/+37
| | | | | | | | | | | | | What populate prints names paths on the target in different ways: populating as host:path, also for the local target, pruning and the errors about the sentinel by targetName, and replacing a symlink or a Git work tree by the path alone. All of them use targetName now: the path, prefixed by the host if the target isn't local, and by the user if that isn't root. Replacing a symlink or a Git work tree quotes the path and ends with an ellipsis, like populating and pruning do. The tests expect the names that way, and rsync-log also expects the line naming the source being populated.
* populate: reduce log spamtv9 days1-2/+2
|
* populate: name the target's user in messages unless it's roottv9 days2-8/+17
| | | | | | | Messages naming the target, i.e. pruning's lines and the errors about the sentinel, name the user as well, as user@host:path, unless the user is root, the usual one, so deploying as another user, e.g. with sudo, shows.
* populate prune: name the target by its host, like populate's messagestv9 days2-26/+19
| | | | | | | | | Pruning names the target as user@host:path, like rsync's destination, but nothing else populate prints names the user: the errors about the sentinel name the host, and the commands it runs on the target name the host, with the user as ssh's option. Pruning names the target like those do, as host:path, and the errors about the sentinel use the same function for it.
* populate prune: name the target like populating doestv9 days2-9/+17
| | | | | | | Pruning prints the path of each entry it removes, without the target's user and host, while rsync, when populating, names the target as user@host:path. Pruning names it the same way, so the lines say which machine they're about.
* tests: check the canaries' modification times and new neighbourstv9 days1-5/+9
| | | | | | | The canary manifest records modification times, so changing a canary's time shows, as rsync does to a hard-linked file it takes for unchanged. And entries beside the target directories whose names start like the canaries' count as canaries, so a file created next to one shows, too.
* populate: fail on inputs drawn that the flake doesn't havetv9 days4-3/+125
| | | | | | | | | | | | | | | Nix ignores an input that --override-input names but the flake doesn't have, with a warning, so a misspelt input name in inputs draws nothing, and the source keeps using its own input. Before changing anything, populate locks the flake of each source that draws inputs, with an empty flake in place of each input drawn, which may be unreachable, and fails, naming the source and the input, unless the flake has each input drawn. Below an input drawn from a source, the path continues in the flake of that source. An edge case misspells an input of the source's flake, and one below an input drawn from a source, both ways of drawing inputs.
* populate: never write through files or change hard links elsewheretv9 days3-13/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | Populate writes some files in place, and makes files writable, which reaches beyond the target directory where such a file is a symlink or a hard link to one elsewhere: - The key of a derived copy gets written to a new file, which then replaces it, instead of written through a symlink at its place. - Lock files get copied to a new file before Nix rewrites them, instead of to flake.lock.populate, which a source may contain as a symlink. - Before rsync syncs a source, each file with other hard links gets replaced by a copy of its own. rsync fixes the mode and time of a file it takes for unchanged in place, which changes every hard link to it: one elsewhere, e.g. in /etc/nixos, or a backup's. - Removing state makes only directories writable, which is what removing what's in them needs, instead of everything, including files that are hard links to ones elsewhere. - The sentinel file only gets created if it's missing, instead of touched on every run. - A link only counts as pointing into the store if its resolved path is in the store, so a link like /nix/store/../.. gets replaced. Safety scenarios cover a symlink in place of a key, symlinks in a source named like the lock file's copies, a hard link to a read-only file outside in the target directory, and in stale state.
* populate git: refuse work trees whose .git names places elsewheretv9 days4-3/+194
| | | | | | | | | | | | | | | | | | | | | | | | | | | | A .git directory names places elsewhere in more ways than being a symlink or a file: core.worktree, symlinks to a configuration, to objects, or to refs elsewhere, a commondir file naming another repository, and linked work trees. Git's commands follow them, so fetching, resetting, setting the remote's URL, and cleaning change what is elsewhere: with core.worktree, git clean removes everything in the work tree it names. And Git runs commands the configuration names, like core.fsmonitor, and hooks. Populate only uses a work tree whose .git directory has no symlinks, no commondir file, no linked work trees, and only configuration that git clone and populate write, where submodules' repositories may have their work tree in the work tree. Otherwise populating fails with an error telling what's wrong, unless the new option reclone replaces the work tree by a fresh clone. Git runs without hooks and without fsmonitor in any case. The work tree also gets reset to the commit if HEAD is there already: a lock file rewritten in place, or created and staged by Nix, stays otherwise once the source stops drawing inputs. A safety scenario refuses work trees with core.worktree, a symlinked configuration, a commondir file, and core.fsmonitor, replaces them with reclone, and checks that a hook doesn't run. A git scenario stops drawing inputs and checks that the work tree is clean.
* tests: move combinations no code path tells apart to exhaustive teststv9 days3-21/+72
| | | | | | | | | | | | | | | | | | | | | The matrices cross every option of a source type with every other one, also where the code handling one never looks at the other. Scenarios for such combinations are exhaustive, and run in their own tests, which nix-build tests doesn't build, but nix-build tests -A exhaustive does: - file and git: drawing inputs combined with rsync's or Git's options, since drawing inputs happens after delivery. - flake: materialize combined with how the flake gets there, except for flakes in Git work trees and in subdirectories, since materializing copies what got delivered. - flake-input: materialize for inputs other than dep. - deploy: /etc/nixos/flake.nix combined with deploying a flake, since it only matters without one. - edge-cases: generated flakes where only delivery matters, and remote targets where drawing inputs alone matters, which happens on the target alike for every kind of target. That's 181 of 707 scenarios.
* tests: check output, test rsync versions, close gaps in what's assertedtv9 days6-24/+207
| | | | | | | | | | | | | | | | | | | | | | Several checks would pass with the feature they cover broken, or cover it only by chance: - Phases can state what a succeeding populate prints. That covers the warnings of insecureLinks, the lines pruning prints, the one about a replaced symlink, and what rsync logs about each item. - The insecureLinks scenarios take their expectations from the version of rsync, so with an rsync older than 3.5.0 they pass without the check ever running. New tests, rsync-3-4-1 and rsync-3-5-0, run them with these versions, which tests/rsync.nix builds from their release tarballs, with ACLs and extended attributes, which backups use. - Pruning has a leftover upload to remove, and a source that stops having inputs to root leaves no empty directory of roots. - A full clone has every commit from the start, so the fetch for a commit it lacks never runs. A git scenario deploys a commit that appears on the server after cloning. - The jq test also runs the Nix function that finds a flake's subdirectory on the target, over both shapes of nix flake metadata's output.
* populate: draw inputs below drawn inputs at any depthtv9 days2-16/+65
| | | | | | | | | | | | | | | | | | | | | An input below an input that gets drawn from a source is an input of the flake in that source. If that flake draws the input on the way from another source itself, the path goes on in the flake of that one, and so forth. Populate follows the path one source deep only, so x.inputs.dep = { follows = "b"; inputs.p.inputs.z.follows = "c"; } has no effect if b draws p from a source: Nix ignores overrides below an input that follows another one, without warning. Populate follows such paths through the sources, for generated flakes as well as for rewritten lock files. Where the way leads through a generated flake, it names the generated flake's inputs, which are its source and the sources that one draws from. So a source with a rewritten lock file can also draw an input below one it draws from a source with a generated flake. Edge cases cover an input two sources deep, both ways of drawing inputs, and a source with a rewritten lock file drawing from one with a generated flake.
* krops writeDeploy: deploy every flake by a URL Nix can't misreadtv9 days2-40/+74
| | | | | | | | | | | | | | | | | | | nixos-rebuild gets generated flakes as path: URLs, but other flakes by their bare paths, which Nix takes for the enclosing Git work tree, if the target directory lies in one. A file source there fails to deploy, because the work tree doesn't track it. Every flake gets deployed by a URL: a Git work tree as git+file:, with submodules if it has any, like populate refers to it, and anything else as path:. The path gets resolved on the build target, because Nix refuses URLs with symlinks on the way, which the target directory may be, and percent-encoded. --build-host and --target-host name no user if the target has none, instead of failing to evaluate. The deploy test also deploys flakes from file and git sources. Its scenarios evaluate without LOGNAME.
* populate: cope with unusual names and pathstv9 days4-57/+154
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Names and paths with characters that mean something to shells, to find, or in URLs get handled as what they are, or refused: - Pruning, and removing state that sources don't need anymore, list entries with find instead of a loop reading names, which strips blanks at the ends of names and splits names at newlines. So an entry named "trail " gets pruned instead of reported as pruned and left. - Pruning doesn't make entries writable before removing them, which would change the mode of files elsewhere that are hard links to files therein. - Source names are portable file names: letters, digits, and ._-, not starting with -. nixpkgs' writers accept nothing else in the names of the scripts populating the sources, which contain the source names, and fail with an assertion that doesn't mention the source. - Paths get percent-encoded where they go into URLs for Nix, which refuses blanks there and takes ? and # for delimiters. - rsync gets the target path protected from the target's shell (--protect-args), which expands wildcards in it otherwise: a file source delivered to /var/src/a?c ends up in /var/src/abc if that exists. A safety scenario deploys every kind of state to a target path with a blank, wildcards, #, and %. An edge case prunes entries with blanks at the ends of their names, with a newline, and with wildcards, and the scenario pruning links also prunes hard links to a read-only file outside, and lists what's left in the C locale, where ls sorts .populate first. The scenario with unusual source names also draws inputs from a source named self.