| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Populate replaces what's at the target paths of sources, removes what
sources don't deliver anymore, and prunes. Nothing of that may reach
beyond the target directory, whatever names sources have and whatever the
target directory contains:
- Source names that aren't file names in the target directory don't
evaluate: the empty name, ., .., .populate, and names with slashes or
control characters. E.g. a git source named "" removes the target
directory, and a file source named .. syncs into its parent.
- Target paths that aren't absolute, that are the root directory, or that
contain .. don't evaluate.
- A symlink at the target path of a source gets replaced, not followed,
and populate says so. rsync-based and flake sources do that already.
git sources replace the link as well, and also a work tree whose .git is
a symlink or a file naming a repository elsewhere, instead of fetching,
resetting, and cleaning there. derivation sources replace a link that
doesn't point into the store, instead of failing.
- Populate refuses to use the sentinel, or a state directory therein, if
it is a symlink, instead of changing and removing what it points to.
- What populate removes recursively gets removed as a link if it is one:
chmod -R follows a link named on the command line.
- Flake and flake-input sources that get patched, get their lock file
rewritten, or are materialized replace a symlink at their target path
like every source does, and say so.
- Nix rewrites lock files in place, which changes other hard links to
them as well, e.g. in a backup or outside the target directory, and
what a lock file that is a symlink points to. Populate replaces a lock
file by a copy of its own before rewriting it.
The target directory itself may be a symlink. Nix refuses path: URLs with
symlinks on the way, so the flakes step resolves the paths it hands to
Nix.
Backups work on a sentinel that is still a file, which becomes a
directory, like for everything else that keeps state.
The tests create canaries outside the target directories: a file and a
directory beside them, and a directory elsewhere with a read-only
directory and a Git work tree with uncommitted changes. The check of
every phase of every scenario fails if a canary changed, or if a store
path that the target directory links to doesn't match its hash anymore.
A new test, safety, points links at the canaries: at the target path of
every source type, inside what sources replace, in place of populate's
state, and among what it removes and prunes. It also covers the names
and target paths that don't evaluate, a target directory that is a
symlink, and a lock file with another hard link.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sources drawing inputs from other sources get the lock file of their
flake rewritten with nix flake lock --override-input, so the flake in
the source itself uses the other sources, and needs no generated flake
to be found. file and git sources get their lock file rewritten in
place. flake and flake-input sources get delivered as a copy with the
rewritten lock file, which gets added to the store with nix-store
--add-fixed, and the target path links to that. With rewriteLock set
to false, and for symlink sources, the source stays as it is and gets
a generated flake in .populate/flakes/<name>, which writeDeploy's flake
parameter uses.
flake and flake-input sources get two more options:
- patches: patches on the deploying machine, which get uploaded to
.populate/patches/<name> and applied in order with patch -p1 on the
target, in the directory the target path shows.
- materialize: the target path becomes a writable copy instead of a
link into the store.
Sources that get patched, rewritten, or materialized get delivered to
.populate/derived/<name>/original, which also roots the delivered store
path, and the flakes step derives the target path from there. Copies
get made again only if what they derive from changes, which a key in
.populate/derived/<name> records.
The matrices of the file, flake, flake-input, and git tests cover both
ways of drawing inputs, those of flake and flake-input materialize, the
deploy test a source with a rewritten lock file, and features and
edge cases run the scenarios drawing inputs both ways. New features
cover patches, a patch that doesn't apply, patches in a subdirectory,
reusing copies, and materializing.
|
| |
|
|
|
|
|
|
|
|
|
| |
Flake and flake-input sources take the flake reference as attribute url,
like git sources take their repository, and like inputs in flake.nix:
something.flake = "github:owner/something";
otherthing.flake = {
url = "github:owner/otherthing";
inputs.nixpkgs.follows = "nixpkgs";
};
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rsync 3.5.0 opens merge files, including per-directory .rsync-filter
files, walking the path one directory at a time, refusing symlinks that
neither root nor the user running rsync owns, and opening each directory
for reading. It treats the files it can't open that way as missing,
without saying so. That happens, e.g., below a home directory with mode
711, which the user may enter but not read, or behind another user's
symlink. Such a source then got synced without its .rsync-filter rules:
excluded files got transferred, and --delete-excluded deleted nothing.
Before syncing with rsync 3.5.0 or later a source containing .rsync-filter
files, populate asks rsync which of them it reads: a dry run with
--debug=FILTER2. If it reports an existing one as missing, the file
source type's new insecureLinks option decides: with null, the default,
populate fails with an error naming the file; with true, rsync reads such
files anyway (--insecure-links); with false, rsync syncs without them.
Both print a warning.
File scenarios sync a source through a symlink owned by nobody, with each
setting.
|
| |
|
|
|
|
|
|
|
| |
Depending on the version of Nix, lock files name subdirectories as
attribute dir or as query parameter dir of the URL. The VM tests run
one version, so only one shape gets covered there. The jq programs for
flakes move to pkgs/populate/flake-jq.nix, and a new test, jq, runs them
without VMs over lock files of both shapes, with follows, removed and
cyclic follows, and inputs given by relative paths.
|
| |
|
|
|
|
|
| |
The git scenarios deployed only the commit a clone checks out anyway, so
populate never fetched, reset, or updated submodules in a full clone, and
fetchAlways made no observable difference. They deploy HEAD's parent
first now.
|
| |
|
|
|
|
|
|
| |
Deploying the same sources twice doesn't show whether generated flakes get
locked again, and the git scenarios' results are the same whether a Git
work tree is referred to as git+file: or path:. A feature scenario
changes what a source contains between two deploys, and the git
scenarios check the reference.
|
| |
|
|
|
|
|
|
|
| |
Uploads an interrupted run left behind in .populate/flake-uploads only got
removed before the same source uploaded again. Delivering a flake or
flake input removes its upload directory now, and the empty
.populate/flake-inputs once the last roots are gone.
An edge case leaves an upload behind between two deploys.
|
| |
|
|
|
|
|
|
|
|
| |
Generated flakes used source names as input names, but Nix accepts fewer
names there than populate does, e.g. not alt-1.0 or self. Following such
a source failed late, at locking, with an error mentioning neither krops
nor the source. Names Nix doesn't accept get replaced by ones derived
from them.
An edge case follows a source named alt-1.0.
|
| |
|
|
|
|
|
|
|
| |
pipe sources get a fixed modification time, so rsync's quick check took
changed output of the same size, e.g. a new commit hash or token, for
unchanged, and didn't deliver it. pipe sources get synced comparing
checksums now.
An edge case changes a pipe source's output without changing its size.
|
| |
|
|
|
|
|
|
|
|
|
| |
Generated flakes got locked with a bare nix flake lock in their directory,
and writeDeploy deployed them by their path. Nix takes both for the
enclosing Git work tree, if the target directory lies in one, e.g. a home
directory kept in Git, and fails, because generated flakes aren't tracked
there. Both refer to generated flakes as path: now.
An edge case deploys into a Git work tree, and the deploy test covers
deploying generated flakes.
|
| |
|
|
|
|
|
|
|
| |
Generated flakes declared every source they draw inputs from as a flake,
so drawing an input declared with flake = false from a source without
flake.nix failed, although the README says it works. Such sources get
declared with flake = false now.
An edge case draws a non-flake input from a source without flake.nix.
|
| |
|
|
|
|
|
|
|
| |
An override below an input drawn from a source that has a generated
flake itself got placed on that generated flake, which has no such
input, so Nix ignored it with a warning. Generated flakes have their
source as input of the same name, so the override goes below that.
An edge case overrides an input of a source with a generated flake.
|
| |
|
|
|
|
|
|
|
|
| |
Populate stopped the ssh master at the end of its script, which a failure
skips. The master then stays for an hour, with its control socket gone
along with populate's temporary directory. Populate stops it on exit
now, before removing that directory.
An edge case checks that no master remains after a failing and after a
succeeding deploy.
|
| |
|
|
|
|
|
|
| |
nix-store --add-root only replaces symlinks pointing into the store, so a
symlink pointing elsewhere at the target path, e.g. one a symlink source
left there, made delivering a flake fail. Populate removes such links.
An edge case switches a symlink source to a flake source.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
rsync follows a symlink at the target path, so a source switching from the
flake or flake-input type to file, pass, passage, or pipe got written into
the store path the link points to, and one switching from the symlink
type into what that link points to, e.g. another source. Populate
removes a symlink at the target path before syncing with rsync.
The roots in .populate/flake-inputs/<name> of a source that is neither a
flake nor a flake-input source anymore get removed as well.
An edge case switches a flake source to a file source.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A flake-input source delivers an input as its own flake, which
evaluating it uses with its own lock file. Populate, however, took the
input's inputs from the parent flake's lock file: on the target, by
walking the lock file, and on the deploying machine, from the parent's
nix flake archive. That had several consequences:
- An empty follows path, which removes an input, or two inputs following
each other made the walk loop forever.
- Copying left out inputs that follow other inputs, and rejected names
through them, e.g. dep/inner where dep's inner follows another input.
- Inputs given by relative paths, e.g. path:./lib, which have no lock
entry of their own to fetch, broke fetching on the target and
flake-input sources.
- Where the parent overrides an input's inputs, the input's own lock file
pins other versions than the delivered ones.
Populate now resolves the name through the parent's lock file, follows
included and guarded against cycles, to what to fetch: a locked
reference, or the flake itself, which inputs given by relative paths are
part of. It fetches that, on the deploying machine or on the target, and
then the inputs the input's own lock file pins, using nix flake archive.
An input without a lock file comes without its inputs.
New edge cases cover lock files with empty and cyclic follows, names
through follows, the delivered closure, and relative path inputs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A new test, edge-cases, covers paths that work but have no scenario yet:
a flake given by an indirect reference, which the target fetches by its
locked reference; a remote flake in a subdirectory; a flake in a
subdirectory drawing inputs from sources; a dirty Git work tree; a flake
whose lock file needs updating; drawing inputs from a source that isn't
a directory; nested names of missing inputs; several sources sharing one
flake; and an input locked to a path that the target has.
The deploying machine gets a registry entry for the indirect reference.
Resolving it fails if the global registry can't be downloaded, which it
can't in the VM, and disabling the global registry with flake-registry
needs flakes, so the deploying machine enables them, like any deploying
machine using indirect references does. The target doesn't, so checks
for flake support on targets still mean something.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Several weaknesses let checks pass although the feature under test is
broken:
- Nix restores fetched inputs from its fetcher cache, so evaluating a
delivered flake offline after collecting garbage succeeds even if
populate rooted none of its inputs. nix_eval uses an empty cache
directory.
- Scenarios share target directories and stores, so store paths and roots
of earlier scenarios stand in for missing ones, and uploads of flakes
never run in a passing scenario. Scenarios of the flake, flake-input,
and features tests are isolated now: they start without target
directories and with a collected store on both machines.
- Phases expecting evaluation to fail pass on any evaluation failure,
e.g. a misspelt option. Each such phase has a control now: sources
differing only in what makes evaluation fail, which must evaluate.
Further, populate runs with a time limit, so a hang fails its scenario
instead of the whole test, phases can run steps on the deploying machine
before and after, and evaluating the scenarios fails if the remote
target counts as local for whoever evaluates them.
|
| |
|
|
|
|
|
|
|
|
|
| |
nixos-rebuild deploys /etc/nixos/flake.nix instead of NIX_PATH if that
file exists, so deploying anything but a flake fails on such targets.
Without writeDeploy's flake parameter, nixos-rebuild gets --no-flake if
/etc/nixos/flake.nix exists on the build target. nixos-rebuild versions
predating flakes neither know --no-flake nor look for that file, so they
don't get it.
The deploy VM test covers targets with and without that file.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
For targets using sudo, writeDeploy runs nixos-rebuild on the target via
sudo, so --use-remote-sudo has no effect there. It only matters when
nixos-rebuild runs on a separate build target and activates the system
on the target, so writeDeploy passes it only then.
A new VM test, deploy, runs writeDeploy with a stand-in for
nixos-rebuild, which records the user it runs as, NIX_PATH, and its
arguments, for deploying flakes and other sources, with and without a
separate build target, to every kind of target.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Populate never removes what it delivered for sources that aren't
configured anymore. For flake and flake-input sources, that keeps their
garbage collector roots, so what they delivered never gets collected.
With prune, populate removes everything from the target directory that
isn't a source, except .populate, after syncing the sources. It also
removes the roots in .populate/flake-inputs of names that aren't flake or
flake-input sources, and uploads left behind by interrupted runs.
Pruning is opt-in, because the target directory may contain other things
worth keeping, or be shared by deployments with different sources.
writeDeploy, writeTest, and writeCommand pass prune on to populate.
A feature scenario deploys sources, deploys fewer without pruning, then
with pruning, and checks that the removed sources' entries and roots are
gone, while the remaining sources and .populate stay.
|
|
|
tests/default.nix provides one VM test per source type, and one for
features spanning sources. Each test runs scenarios from
tests/scenarios.nix, which cover every combination of a source type's
options, deployed to the deploying machine itself, to a remote target as
root, and to a remote target using sudo. A scenario prepares the target
directory, deploys, checks the result, and deploys and checks again.
Scenarios expecting populate to fail check its error message, or check
at evaluation time if populate is expected not to evaluate.
The deploying machine creates fixtures using tests/fixtures.sh and serves
them via HTTP and git daemon, so flakes can be fetched on the target
without further network access. Checks evaluate delivered flakes without
network access to the fixtures after collecting garbage, so they verify
that the delivered store paths are complete and rooted.
|
