1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
|
# Populate's design
Populate delivers sources to a target directory. It consists of two parts:
- Nix code, which turns populate's arguments into a spec: the sources as
`evalSource` returns them, the target, and the options, as JSON. It
checks the spec during evaluation, so mistakes show early.
- A program, which reads a spec and delivers the sources. It knows
nothing about Nix expressions: whatever produces a spec can drive it.
So there is one program for each version of populate, instead of a script
for each set of arguments, and the program can be written in any language.
This directory holds the program in Rust. Another implementation, e.g. in
Haskell, reads the same spec, prints the same messages, and passes the same
tests.
`shell.nix` is the shell implementation, which generates a script for each
set of arguments. It stays until the program delivers everything it does,
and defines the behaviour the program has to match.
## Nix side
`default.nix` is `pkgs.populate`. Given populate's arguments, `{ backup ?
false, force ? false, prune ? false, source, target }`, it returns the
command delivering them. `implementation` selects how:
- `"shell"`, the default: `shell.nix`.
- `"rust"`: `rust.nix`, which checks the arguments with `checks.nix`, turns
them into a spec, and returns a script running the program with the spec
on file descriptor 3, so the commands populate runs keep standard input.
The script puts the tools populate runs in front of `PATH`, and depends
on the store paths in the spec, so building it builds them. `rustProgram` is the program, by default built by `program.nix`.
`writeDeploy` and the other functions of `pkgs.krops` use `pkgs.populate`,
so their arguments stay as they are.
## Invocation
```
populate SPEC
```
`SPEC` is the path of a file containing the spec, or `-` for standard
input. Exit status:
- 0: populate delivered all sources.
- 1: a check of the spec failed, or delivering did.
- 2: the arguments are wrong, or the spec can't be read, isn't JSON, is of
another version, or doesn't have the spec's shape.
- 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped.
## Spec, version 1
The spec is a JSON object:
| Field | Type | Default | Meaning |
|-----------|----------------------|---------|-----------------------------------------------|
| `version` | number | | The version of the spec: 1. |
| `backup` | boolean | false | Copy the sources to backup first. |
| `force` | boolean | false | Create the sentinel if it's missing. |
| `prune` | boolean | false | Remove what isn't a source. |
| `source` | object of sources | | The sources, by name. |
| `target` | object | | Where to deliver them. |
`target` has these fields:
| Field | Type | Default | Meaning |
|----------------|------------------|---------|------------------------------------------|
| `user` | string or null | null | The user to log in as. |
| `host` | string | | The host to deliver to. |
| `port` | string or null | null | The port ssh connects to. |
| `path` | string | | The target directory. |
| `sudo` | boolean | false | Deliver as root, via sudo. |
| `extraOptions` | array of strings | [] | More arguments for ssh. |
| `local` | boolean or null | null | Whether the target is local. |
Each source is an object with the field `type`, the source's type, and a
field named after the type, holding the source's settings:
```json
{ "type": "git", "git": { "url": "https://...", "ref": "main", ... } }
```
The types and their settings are those of `lib/types/populate.nix`, as
`evalSource` returns them: with shorthands expanded, e.g. `file =
"/etc/x"` as `{ "path": "/etc/x" }`, and without settings that are null.
The source types are `derivation`, `file`, `flake`, `flake-input`, `git`,
`pass`, `passage`, `pipe`, and `symlink`.
Fields the spec doesn't define are errors, so a misspelt option doesn't go
unnoticed, and a spec of another version gets refused before anything
else. A new version of the spec adds fields or changes their meaning; the
program reads the versions it knows.
Store paths in the spec, e.g. of a `file` source that is a derivation, must
exist when the program runs. The script `rust.nix` writes depends on them.
Something producing a spec at run time has to build them first.
## Checks
The program checks the spec as `checks.nix` does, with the same messages:
- Source names are portable file names: letters, digits, and `._-`, not
starting with `-`, and neither `.`, `..`, nor `.populate`.
- The target path is absolute, not the root directory, and contains
neither `..` nor control characters.
- Inputs get drawn from sources that exist, not from the source itself,
and not in circles.
Checks that depend on where populate runs come on top, e.g. refusing to
copy flakes to targets using sudo, which needs to know whether the target
is local.
## Output
What populate prints on standard error is part of its interface: people
read it, and the tests check it. Messages are those of `shell.nix`, e.g.:
```
populating 'host:/var/src/nixpkgs'...
pruning 'host:/var/src/old'...
replacing symlink 'host:/var/src/x', which points to '/elsewhere'...
replacing the Git work tree 'host:/var/src/x', because ...
```
A target path in a message starts with `user@host:` for remote targets,
without `user@` if the user is root or not set, and is a plain path for
the local target. Errors start with `error:` or `krops:`, warnings with
`warning:`. rsync's itemized changes appear as `%i %n%L %b/%l`.
## Safety
Populate never changes or removes anything outside the target directory,
whatever the spec and whatever it finds on the target:
- The checks above keep source names and the target path inside it.
- A symlink at the target path of a source gets replaced, never written
through, and populate's state directory, `.populate`, must not be one.
- Removing a tree makes only its directories writable, and removes what's
below them, not what symlinks point to.
- Hard links to files of a source get broken before rsync writes to them.
- A Git work tree gets reused only if its `.git` is a directory of its
own, with no other configuration than `git clone` and populate set,
and Git runs without hooks and file system monitors.
- Temporary files get created with `mktemp`, and moved into place.
The tests' canaries, files around the target directories and in other
places populate must not touch, check this after every phase.
## Execution
Whether a target is local is what the target's `local` says, or, if
that's null, gets decided when the program runs: it is, if its user is
`LOGNAME` and its host is `localhost` or the host's name, from `HOSTNAME`
or `/etc/hostname`. `rust.nix` decides during evaluation, like the other
functions of krops. For a local target, populate runs commands directly.
For a remote target, populate runs commands over ssh, sharing one master
connection per target. The commands are fixed POSIX shell snippets, with
values passed as positional arguments, so nothing gets quoted into them.
The target needs a POSIX shell, coreutils, findutils, rsync, Git for
`git` sources, and Nix for flakes; fresh installations of NixOS lack rsync
and Git. With `sudo`, the snippets run via `sudo`.
Populate runs rsync, Git, ssh, and Nix as programs, found in `PATH`.
## Development
- `cargo test` in this directory runs the program's unit tests.
- `nix-build tests --argstr implementation rust -A smoke` runs the smoke
test, a few scenarios covering each source type and each kind of target,
with the program.
- `nix-build tests --argstr implementation rust` runs all scenarios with
the program. A change has to pass them with both implementations, as
long as there are two.
## Status
The program delivers all source types, draws inputs from sources, makes
backups, and prunes. The code for flakes, in `src/flake.rs`, builds the
same shell code that `shell.nix` generates, with every value quoted, so
both implementations behave alike; it moves to Rust piece by piece, with
the scenarios telling whether it still behaves alike.
The jq programs in `flake-jq.nix`, which the shell implementation and the
jq test use, have equivalents in Rust, with the jq test's cases as unit
tests.
|