aboutsummaryrefslogtreecommitdiffstats
path: root/pkgs/populate
Commit message (Collapse)AuthorAgeFilesLines
* populate file: add insecureLinkstv34 hours1-0/+55
| | | | | | | | | | | | | | | | | | | | | | rsync 3.5.0 opens merge files, including per-directory .rsync-filter files, walking the path one directory at a time, refusing symlinks that neither root nor the user running rsync owns, and opening each directory for reading. It treats the files it can't open that way as missing, without saying so. That happens, e.g., below a home directory with mode 711, which the user may enter but not read, or behind another user's symlink. Such a source then got synced without its .rsync-filter rules: excluded files got transferred, and --delete-excluded deleted nothing. Before syncing with rsync 3.5.0 or later a source containing .rsync-filter files, populate asks rsync which of them it reads: a dry run with --debug=FILTER2. If it reports an existing one as missing, the file source type's new insecureLinks option decides: with null, the default, populate fails with an error naming the file; with true, rsync reads such files anyway (--insecure-links); with false, rsync syncs without them. Both print a warning. File scenarios sync a source through a symlink owned by nobody, with each setting.
* populate: log what rsync changes about each itemtv34 hours1-0/+4
| | | | | | rsync logged only the names of the items it transferred. It logs what changed about each item now (--itemize-changes), along with its size as transferred and in total.
* tests: run populate's jq programs over lock files of both shapestv43 hours2-66/+71
| | | | | | | | | Depending on the version of Nix, lock files name subdirectories as attribute dir or as query parameter dir of the URL. The VM tests run one version, so only one shape gets covered there. The jq programs for flakes move to pkgs/populate/flake-jq.nix, and a new test, jq, runs them without VMs over lock files of both shapes, with follows, removed and cyclic follows, and inputs given by relative paths.
* populate flake-input: remove uploads left behindtv43 hours1-0/+5
| | | | | | | | | Uploads an interrupted run left behind in .populate/flake-uploads only got removed before the same source uploaded again. Delivering a flake or flake input removes its upload directory now, and the empty .populate/flake-inputs once the last roots are gone. An edge case leaves an upload behind between two deploys.
* populate: name inputs of generated flakes validlytv43 hours1-7/+14
| | | | | | | | | | Generated flakes used source names as input names, but Nix accepts fewer names there than populate does, e.g. not alt-1.0 or self. Following such a source failed late, at locking, with an error mentioning neither krops nor the source. Names Nix doesn't accept get replaced by ones derived from them. An edge case follows a source named alt-1.0.
* populate pipe: deliver changed output of the same sizetv43 hours1-1/+3
| | | | | | | | | pipe sources get a fixed modification time, so rsync's quick check took changed output of the same size, e.g. a new commit hash or token, for unchanged, and didn't deliver it. pipe sources get synced comparing checksums now. An edge case changes a pipe source's output without changing its size.
* populate, krops writeDeploy: refer to generated flakes as path:tv43 hours1-1/+2
| | | | | | | | | | | Generated flakes got locked with a bare nix flake lock in their directory, and writeDeploy deployed them by their path. Nix takes both for the enclosing Git work tree, if the target directory lies in one, e.g. a home directory kept in Git, and fails, because generated flakes aren't tracked there. Both refer to generated flakes as path: now. An edge case deploys into a Git work tree, and the deploy test covers deploying generated flakes.
* populate: draw inputs from sources that aren't flakestv43 hours1-0/+5
| | | | | | | | | Generated flakes declared every source they draw inputs from as a flake, so drawing an input declared with flake = false from a source without flake.nix failed, although the README says it works. Such sources get declared with flake = false now. An edge case draws a non-flake input from a source without flake.nix.
* populate: place overrides on sources with generated flakes correctlytv43 hours1-0/+3
| | | | | | | | | An override below an input drawn from a source that has a generated flake itself got placed on that generated flake, which has no such input, so Nix ignored it with a warning. Generated flakes have their source as input of the same name, so the override goes below that. An edge case overrides an input of a source with a generated flake.
* populate: stop the ssh master also after failurestv43 hours1-5/+8
| | | | | | | | | | Populate stopped the ssh master at the end of its script, which a failure skips. The master then stays for an hour, with its control socket gone along with populate's temporary directory. Populate stops it on exit now, before removing that directory. An edge case checks that no master remains after a failing and after a succeeding deploy.
* populate flake-input: replace symlinks pointing outside the storetv43 hours1-1/+7
| | | | | | | | nix-store --add-root only replaces symlinks pointing into the store, so a symlink pointing elsewhere at the target path, e.g. one a symlink source left there, made delivering a flake fail. Populate removes such links. An edge case switches a symlink source to a flake source.
* populate: replace symlinks before syncing with rsynctv43 hours1-0/+18
| | | | | | | | | | | | | rsync follows a symlink at the target path, so a source switching from the flake or flake-input type to file, pass, passage, or pipe got written into the store path the link points to, and one switching from the symlink type into what that link points to, e.g. another source. Populate removes a symlink at the target path before syncing with rsync. The roots in .populate/flake-inputs/<name> of a source that is neither a flake nor a flake-input source anymore get removed as well. An edge case switches a flake source to a file source.
* populate flake-input: deliver the inputs the input's own lock file pinstv43 hours1-138/+174
| | | | | | | | | | | | | | | | | | | | | | | | | | | | A flake-input source delivers an input as its own flake, which evaluating it uses with its own lock file. Populate, however, took the input's inputs from the parent flake's lock file: on the target, by walking the lock file, and on the deploying machine, from the parent's nix flake archive. That had several consequences: - An empty follows path, which removes an input, or two inputs following each other made the walk loop forever. - Copying left out inputs that follow other inputs, and rejected names through them, e.g. dep/inner where dep's inner follows another input. - Inputs given by relative paths, e.g. path:./lib, which have no lock entry of their own to fetch, broke fetching on the target and flake-input sources. - Where the parent overrides an input's inputs, the input's own lock file pins other versions than the delivered ones. Populate now resolves the name through the parent's lock file, follows included and guarded against cycles, to what to fetch: a locked reference, or the flake itself, which inputs given by relative paths are part of. It fetches that, on the deploying machine or on the target, and then the inputs the input's own lock file pins, using nix flake archive. An input without a lock file comes without its inputs. New edge cases cover lock files with empty and cyclic follows, names through follows, the delivered closure, and relative path inputs.
* populate: add prunetv45 hours1-1/+40
| | | | | | | | | | | | | | | | | | | Populate never removes what it delivered for sources that aren't configured anymore. For flake and flake-input sources, that keeps their garbage collector roots, so what they delivered never gets collected. With prune, populate removes everything from the target directory that isn't a source, except .populate, after syncing the sources. It also removes the roots in .populate/flake-inputs of names that aren't flake or flake-input sources, and uploads left behind by interrupted runs. Pruning is opt-in, because the target directory may contain other things worth keeping, or be shared by deployments with different sources. writeDeploy, writeTest, and writeCommand pass prune on to populate. A feature scenario deploys sources, deploys fewer without pruning, then with pruning, and checks that the removed sources' entries and roots are gone, while the remaining sources and .populate stay.
* populate: stop the ssh master only if there is onetv47 hours1-1/+4
| | | | | | | Populate ends by stopping the ssh master, which fails if no source started one, e.g. for local targets, which populate doesn't reach via ssh. Populate then exits with ssh's exit status, though all sources got delivered. Populate now checks for a master first.
* populate flake-input: support subdirectories named in URLstv47 hours1-7/+41
| | | | | | | | Depending on the version of Nix, flake references and lock entries name the subdirectory containing a flake either as attribute dir or as query parameter dir of their URL. fetchTree supports neither, so populate looks for both: it takes the subdirectory from either, from the locked or the original reference, and removes both before fetching.
* populate flake: support fetchOnTargettv2 days1-22/+183
| | | | | | | | | | | | | | | | | | With fetchOnTarget, the target fetches the flake's inputs using the arguments recorded in the lock file, like it does for flake-input sources. The flake itself has no lock entry, so the target fetches it by its locked reference, unless that refers to the deploying machine, e.g. a path or a Git work tree. Then the flake's tree gets transferred using rsync and added to the target's store using nix-store --add-fixed, which yields the same store path as on the deploying machine, because flake store paths are content-addressed by their NAR hash. Adding paths like this needs neither signatures nor a trusted user, so this works for targets using sudo. The transfer gets skipped if the target's store has the flake already, and populate fails if the added path differs from the expected one. rsync' gets a filterFiles option, so the flake's tree gets transferred without applying .rsync-filter files, which would change it.
* populate: add flake-input source typetv2 days1-9/+11
| | | | | | | The flake-input source type delivers an input of a flake, as recorded in the flake's lock file, and the flake source type delivers the flake itself. fetchOnTarget belongs to flake-input, because only a locked input can be fetched on the target.
* populate: generate flakes for inputs drawn from sourcestv2 days1-5/+131
| | | | | | | | | | | | | | For each source that draws inputs from other sources, populate generates a flake in .populate/flakes/<name>, which re-exports the outputs of the source's flake with those inputs drawn from the other sources. The source itself stays as it is, so NIX_PATH still refers to it. Generated flakes refer to each other's generated flakes, so a source's own declarations apply wherever it gets drawn from, and get generated in dependency order. Evaluation fails for sources drawing inputs from missing sources, from themselves, or from each other. Generated flakes get locked on every run, and get removed once their sources no longer draw inputs from other sources.
* populate flake: add fetchOnTargettv2 days1-18/+80
| | | | | | | | | | With fetchOnTarget, the target fetches the input and its own inputs itself, using fetchTree with the arguments recorded in the flake's lock file. Only nix flake metadata runs on the deploying machine, so the inputs get downloaded once, by the target. This needs Nix with flake support on the target, which populate checks before fetching. Because nothing gets copied, this also works for targets using sudo.
* populate: add flake source typetv2 days1-1/+97
| | | | | | | | | | | | Deliver a flake, or one of its locked inputs, to the target: archive the flake locally, copy the selected input together with its own inputs into the target's store, and link it at the target path, registered as garbage collector root. The result can be used in NIX_PATH as well as by flakes evaluated on the target, without network access. nix copy runs with a separate ssh wrapper without the control socket, because Nix waits for its LocalCommand, which may not run over a multiplexed connection.
* populate: export TMPDIRtv2 days1-0/+1
| | | | | | | Each job and the ssh wrapper run as separate scripts, which only see TMPDIR if it's exported. Without TMPDIR in the caller's environment, jobs fail, and the ssh wrapper's control socket ends up at the root directory.
* populate backup: fix error messagetv2 days1-1/+1
|
* populate: derive socket name from target hosttv2026-01-011-2/+2
|
* populate: use ssh control socket1.30.0tv2025-12-091-6/+12
|
* populate use singular tmpdirtv2025-12-091-24/+22
| | | | | | | | Set TMPDIR to a fresh, private top-level temporary directory, and run each job in a separate subdirectory beneath it. TMPDIR remains shared across all jobs, allowing reuse of resources like ssh control sockets.
* populate pass{,age}: use $pass_outtv2025-12-091-9/+9
|
* populate pipe: use rsynctv2025-12-091-2/+10
| | | | Generate a reproducible output to transmit it only if it has changed.
* populate passage: rename loop variablestv2025-04-091-6/+7
|
* populate rsync': reduce verbositytv2025-04-091-1/+2
|
* populate: add passage source type1.29.0tv2024-01-111-1/+32
|
* populate pass: calculate hash for each symlink1.28.2lassulus2023-03-131-1/+9
| | | | | this fixes folders with symlinks not getting updated if the symlinked file is in another subfolder of the pass repo
* populate git: remove trailing spaces1.26.3tv2022-02-121-1/+1
|
* populate git: remove non-worktree targettv2022-02-121-0/+5
|
* target: use default port from SSH configErik Arvstedt2021-11-201-2/+1
| | | | | This is the expected behavior. The SSH config is also implicitly used for other SSH-related settings.
* populate file: isDerivation -> isStorePath1.26.1tv2021-10-261-2/+2
|
* populate file: admit derivations1.26.0tv2021-10-261-4/+15
|
* populate: add rsyncDefaultConfigtv2021-10-261-7/+14
|
* populate pass: don't decrypt .gpg-idlassulus2021-02-161-1/+1
|
* populate pass: make git optional againlassulus2021-01-171-2/+6
|
* Make pass source follow symlinksv1.24.0Simon Bruder2020-12-151-1/+1
|
* add `shallow` option to git source typeEnno Lohmeier2020-07-021-3/+15
|
* treewide: harmonize ssh callsv1.22.0tv2020-06-081-12/+11
|
* add target.extraOptionsErik Arvstedt2020-04-181-0/+1
|
* Merge pull request #15 from nyantec/feature/filtersv1.20.0tv2020-02-181-2/+5
|\ | | | | support for include filters
| * support for include filtersMilan Pässler2020-02-031-2/+5
| | | | | | | | | | | | | | - Added a new option for `file` sources, which allows to specify a list of `include` and `exclude` filters - Added a new option for `file` sources, which allows to disable the `--delete-excluded` behaviour.
* | use writers from nixpkgsJörg Thalheim2020-02-171-3/+3
|/ | | | | This makes the evaluation of krops pure (no import from derivation) and makes it faster since the fetchGit result might be garbage collected.
* populate: re-add default values for useChecksum/excludeJörg Thalheim2019-12-231-2/+2
| | | | Turns out, those where required.
* populate file: add exclude optionJörg Thalheim2019-12-231-2/+5
|
* ssh: support using ssh-configured usertv2019-11-291-5/+7
| | | | | | | Let ssh use the user configured in its configuration files when the target user is set to the empty string. Closes https://github.com/krebs/krops/issues/2
generated by cgit v1.3.1 (git 2.54.0) at 2026-11-13 21:26:52 +0000