| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The shell doesn't run the EXIT trap when a signal ends it, so an
interrupted deploy leaves the ssh master, for an hour, and populate's
temporary directory. Populate traps HUP, INT, and TERM and exits, which
runs the EXIT trap.
The shell runs a trap once the command it waits for ends. A signal to
the process group, like the one of an interrupt key, of a closed terminal,
or of timeout, ends that command as well. A signal to populate alone
takes effect after the step that is running.
Phases of scenarios can end populate after some seconds. An edge case
does so while a pipe source's command sleeps, and checks that neither the
ssh master nor the temporary directory remain.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rsync reads .rsync-filter files in the directories above a source as
well, and their rules apply to the transfer. The check for files that
rsync 3.5.0 and later can't read only starts if the source itself
contains one, so a subdirectory of a repository with a filter file at
its top gets synced without it, silently. The check also starts for
filter files above the source.
The check's dry run is the transfer it stands for, as far as filtering
goes:
- It gets the source's exclude and filters, so it doesn't enter
directories the transfer skips.
- It doesn't run for transfers that apply no .rsync-filter files, i.e.
uploads of flakes and patches.
- A filter file the user may not read anyway doesn't count: rsync fails
on its own when it gets to transfer that file, and --insecure-links
changes nothing about it.
- rsync prints bytes it takes for unprintable as \#ooo, which depends on
the locale. File names get decoded before checking that they exist, so
files below, e.g., directories with umlauts in their names don't pass
for missing in the C locale.
- If rsync doesn't tell which filter files it reads at all, populate
warns that it can't tell either.
The README mentions that --insecure-links also affects the way to the
target path on local targets, and that only rsync 3.5.0 ignores filter
files below directories that may be entered but not read.
Phases of scenarios can set environment variables for populate. New
file scenarios sync a subdirectory whose filter file is above it, and a
source with a filter file below a directory with an umlaut in its name,
in the C locale.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
inputs
Delivering flakes and flake inputs gets more robust:
- With fetchOnTarget, the inputs get collected before the pipeline that
links them, so populate fails if the target can't fetch one, instead of
linking the flake without its inputs and succeeding.
- What the lock file pins gets taken from the store if it's there: the
store path is the one of a directory named source with the locked hash,
which nix-store --print-fixed-path computes. builtins.fetchTree
contacts where the input comes from even then, so redeploying unchanged
sources fails without that if a server is unreachable.
- With fetchOnTarget, a delivered input that is itself locked to a local
path the target lacks fails with populate's error naming the input, like
its inputs do.
- Inputs that get drawn from other sources are left out, with all they
pin: Nix archives an empty flake in their place. So nothing gets
fetched, copied, and rooted that no evaluation uses, and a pin that is
unreachable doesn't matter if it gets replaced anyway.
- flake-input sources get the option requireInputs. By default,
populating fails if the inputs that the input's own lock file pins
can't be delivered, e.g. because the lock file lacks an input its
flake.nix has. With false, populate warns and delivers the input with
what Nix locks now, or else without its inputs.
A remote flake that the target fetches entirely on its own collects its
inputs like everything else does. The Nix function finding its
subdirectory moves next to the jq programs.
New edge cases cover each of these.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sources drawing inputs from other sources get the lock file of their
flake rewritten with nix flake lock --override-input, so the flake in
the source itself uses the other sources, and needs no generated flake
to be found. file and git sources get their lock file rewritten in
place. flake and flake-input sources get delivered as a copy with the
rewritten lock file, which gets added to the store with nix-store
--add-fixed, and the target path links to that. With rewriteLock set
to false, and for symlink sources, the source stays as it is and gets
a generated flake in .populate/flakes/<name>, which writeDeploy's flake
parameter uses.
flake and flake-input sources get two more options:
- patches: patches on the deploying machine, which get uploaded to
.populate/patches/<name> and applied in order with patch -p1 on the
target, in the directory the target path shows.
- materialize: the target path becomes a writable copy instead of a
link into the store.
Sources that get patched, rewritten, or materialized get delivered to
.populate/derived/<name>/original, which also roots the delivered store
path, and the flakes step derives the target path from there. Copies
get made again only if what they derive from changes, which a key in
.populate/derived/<name> records.
The matrices of the file, flake, flake-input, and git tests cover both
ways of drawing inputs, those of flake and flake-input materialize, the
deploy test a source with a rewritten lock file, and features and
edge cases run the scenarios drawing inputs both ways. New features
cover patches, a patch that doesn't apply, patches in a subdirectory,
reusing copies, and materializing.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rsync 3.5.0 opens merge files, including per-directory .rsync-filter
files, walking the path one directory at a time, refusing symlinks that
neither root nor the user running rsync owns, and opening each directory
for reading. It treats the files it can't open that way as missing,
without saying so. That happens, e.g., below a home directory with mode
711, which the user may enter but not read, or behind another user's
symlink. Such a source then got synced without its .rsync-filter rules:
excluded files got transferred, and --delete-excluded deleted nothing.
Before syncing with rsync 3.5.0 or later a source containing .rsync-filter
files, populate asks rsync which of them it reads: a dry run with
--debug=FILTER2. If it reports an existing one as missing, the file
source type's new insecureLinks option decides: with null, the default,
populate fails with an error naming the file; with true, rsync reads such
files anyway (--insecure-links); with false, rsync syncs without them.
Both print a warning.
File scenarios sync a source through a symlink owned by nobody, with each
setting.
|
| |
|
|
|
|
|
|
| |
Deploying the same sources twice doesn't show whether generated flakes get
locked again, and the git scenarios' results are the same whether a Git
work tree is referred to as git+file: or path:. A feature scenario
changes what a source contains between two deploys, and the git
scenarios check the reference.
|
| |
|
|
|
|
|
|
|
| |
pipe sources get a fixed modification time, so rsync's quick check took
changed output of the same size, e.g. a new commit hash or token, for
unchanged, and didn't deliver it. pipe sources get synced comparing
checksums now.
An edge case changes a pipe source's output without changing its size.
|
| |
|
|
|
|
|
|
|
| |
Generated flakes declared every source they draw inputs from as a flake,
so drawing an input declared with flake = false from a source without
flake.nix failed, although the README says it works. Such sources get
declared with flake = false now.
An edge case draws a non-flake input from a source without flake.nix.
|
| |
|
|
|
|
|
|
|
| |
An override below an input drawn from a source that has a generated
flake itself got placed on that generated flake, which has no such
input, so Nix ignored it with a warning. Generated flakes have their
source as input of the same name, so the override goes below that.
An edge case overrides an input of a source with a generated flake.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A flake-input source delivers an input as its own flake, which
evaluating it uses with its own lock file. Populate, however, took the
input's inputs from the parent flake's lock file: on the target, by
walking the lock file, and on the deploying machine, from the parent's
nix flake archive. That had several consequences:
- An empty follows path, which removes an input, or two inputs following
each other made the walk loop forever.
- Copying left out inputs that follow other inputs, and rejected names
through them, e.g. dep/inner where dep's inner follows another input.
- Inputs given by relative paths, e.g. path:./lib, which have no lock
entry of their own to fetch, broke fetching on the target and
flake-input sources.
- Where the parent overrides an input's inputs, the input's own lock file
pins other versions than the delivered ones.
Populate now resolves the name through the parent's lock file, follows
included and guarded against cycles, to what to fetch: a locked
reference, or the flake itself, which inputs given by relative paths are
part of. It fetches that, on the deploying machine or on the target, and
then the inputs the input's own lock file pins, using nix flake archive.
An input without a lock file comes without its inputs.
New edge cases cover lock files with empty and cyclic follows, names
through follows, the delivered closure, and relative path inputs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A new test, edge-cases, covers paths that work but have no scenario yet:
a flake given by an indirect reference, which the target fetches by its
locked reference; a remote flake in a subdirectory; a flake in a
subdirectory drawing inputs from sources; a dirty Git work tree; a flake
whose lock file needs updating; drawing inputs from a source that isn't
a directory; nested names of missing inputs; several sources sharing one
flake; and an input locked to a path that the target has.
The deploying machine gets a registry entry for the indirect reference.
Resolving it fails if the global registry can't be downloaded, which it
can't in the VM, and disabling the global registry with flake-registry
needs flakes, so the deploying machine enables them, like any deploying
machine using indirect references does. The target doesn't, so checks
for flake support on targets still mean something.
|
|
|
tests/default.nix provides one VM test per source type, and one for
features spanning sources. Each test runs scenarios from
tests/scenarios.nix, which cover every combination of a source type's
options, deployed to the deploying machine itself, to a remote target as
root, and to a remote target using sudo. A scenario prepares the target
directory, deploys, checks the result, and deploys and checks again.
Scenarios expecting populate to fail check its error message, or check
at evaluation time if populate is expected not to evaluate.
The deploying machine creates fixtures using tests/fixtures.sh and serves
them via HTTP and git daemon, so flakes can be fetched on the target
without further network access. Checks evaluate delivered flakes without
network access to the fixtures after collecting garbage, so they verify
that the delivered store paths are complete and rooted.
|
