aboutsummaryrefslogtreecommitdiffstats
path: root/pkgs/populate/docs/design.md
blob: 940ab5fadb7bb1818f03e0b3611a8078b93c534f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
# Populate's design

Populate delivers sources to a target directory.  It consists of:

- A program, which reads the sources as JSON and delivers them to the
  target its command line names.  It knows nothing about Nix expressions:
  whatever produces sources as JSON can drive it.
- A Nix library, `lib`, with the types of sources, which describe how to
  write them in Nix, and `evalSource`, which turns sources written so into
  the JSON the program reads.

krops uses both: its `pkgs.populate` turns populate's arguments into a
command line, and checks them during evaluation, so mistakes show early.
So there is one program for each version of populate, instead of a script
for each set of arguments, and the program can be written in any language.
This directory holds the program in Rust.  Another implementation, e.g. in
Haskell, takes the same command line, prints the same messages, and passes
the same tests.

## Nix side

`default.nix` builds the program: `nix-build` in this directory, or
`callPackage`, as krops' `pkgs.populate-unwrapped`.  `lib/default.nix` is
the Nix library, taking nixpkgs' lib.

krops' `pkgs.populate`, in its `pkgs/krops/populate.nix`, is populate's Nix
API there.  Given populate's arguments, `{ backup ? false, force ? false,
prune ? false, source, target }`, it checks them with krops'
`pkgs/krops/checks.nix`, and returns a script running the program with the
sources as JSON on file descriptor 3, so the commands populate runs keep
standard input.  A target string goes to the program as it is.  A target
attribute set, as `mkTarget` returns it, goes part by part, as options,
with `--local` or `--remote` as krops' lib's `isLocalTarget` decides
during evaluation, like the other functions of krops.  The script puts the
tools populate runs in front of `PATH`, and depends on the store paths in
the sources, so building it builds them.  krops' `writeDeploy` and its
other functions use `pkgs.populate`, so their arguments stay as they are.

## Invocation

```
populate [OPTION]... SOURCES [TARGET]
```

`SOURCES` is the path of a file containing the sources, or `-` for
standard input.  `TARGET` is `[USER@][HOST][:PORT][PATH]`, parsed like
krops' lib's `mkTarget` parses it: the user up to the first `@`, the host
up to a `:` or a `/`, the port up to a `/`, and the path from there.  What
it leaves out gets filled in when the program runs, as `mkTarget` would:
the user from `LOGNAME`, the host with the machine's name, and the path
with `/var/src`.  An empty user means ssh's default.

| Option          | Meaning                                                 |
|-----------------|---------------------------------------------------------|
| `--backup`      | Copy the target directory to a backup first.            |
| `--force`       | Create the target directory's sentinel if it's missing. |
| `--prune`       | Remove what isn't a source from the target directory.   |
| `--sudo`        | Deliver as root, via sudo on the target.                |
| `--ssh-arg ARG` | Pass `ARG` to ssh; repeatable.                          |
| `--user USER`   | The target's user, instead of `TARGET`'s.               |
| `--host HOST`   | The target's host, e.g. one `TARGET` can't express.     |
| `--port PORT`   | The port ssh connects to.                               |
| `--path PATH`   | The target directory.                                   |
| `--local`       | The target is the machine populate runs on.             |
| `--remote`      | The target isn't, even if it looks so.                  |
| `-h`, `--help`  | Print the usage and exit.                               |
| `-V`, `--version` | Print populate's version and exit.                    |

Options take their argument as the next argument or after `=`, and may
come before, between, or after `SOURCES` and `TARGET`, up to `--`.  Exit
status:

- 0: populate delivered all sources.
- 1: a check of the sources or the target failed, or delivering did, with
  a message of its own.
- The exit status of the command that failed, if delivering failed with
  that, like the shell implementation does: e.g. 255 if ssh can't reach
  the target, or 128 if git fetch fails.
- 2: the command line is wrong, or the sources can't be read, aren't JSON,
  or aren't sources.
- 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped.

## Sources

The sources are a JSON object, of sources by name.  Each source is an
object with the field `type`, the source's type, and a field named after
the type, holding the source's settings:

```json
{ "nixpkgs": { "type": "git", "git": { "url": "https://...", "ref": "main" } } }
```

The types and their settings are those of `lib/types/populate.nix`, as
`evalSource` returns them: with shorthands expanded, e.g. `file =
"/etc/x"` as `{ "path": "/etc/x" }`, and without settings that are null.
Settings with defaults may be left out.  The source types are
`derivation`, `file`, `flake`, `flake-input`, `git`, `pass`, `passage`,
`pipe`, and `symlink`.

Fields and settings populate doesn't know are errors, so a misspelt option
doesn't go unnoticed.  The sources carry no version: this is version 0, and
a later version of populate that reads them differently will mark them.

Store paths in the sources, e.g. of a `file` source that is a derivation,
must exist when the program runs.  The script krops' `pkgs.populate`
writes depends on them.  Something producing sources at run time has to build them first.

## Checks

The program checks the sources and the target as krops' `checks.nix` does
during evaluation, with the same messages:

- The target's host isn't empty, and neither it nor its user starts with
  `-`, so ssh can't take them for options.  `checks.nix` doesn't check
  this, since krops' targets come from the configuration.
- Source names are portable file names: letters, digits, and `._-`, not
  starting with `-`, and neither `.`, `..`, nor `.populate`.
- The target path is absolute, not the root directory, and contains
  neither `..` nor control characters.
- Inputs get drawn from sources that exist, not from the source itself,
  and not in circles.
- The settings of a source are those `lib/types/populate.nix` defines for
  its type, of the kinds it defines, e.g. absolute paths for the paths of
  `file`, `pass`, and `pipe` sources.  `checks.nix` doesn't repeat this,
  since `evalSource` checks the types.

Checks that depend on whether the target is local come on top, e.g.
refusing to copy flakes to remote targets using sudo.  krops' `checks.nix`
does that during evaluation too, where `pkgs.populate` decides whether the
target is local.

## Output

What populate prints on standard error is part of its interface: people
read it, and the tests check it.  For example:

```
populating 'host:/var/src/nixpkgs'...
pruning 'host:/var/src/old'...
replacing symlink 'host:/var/src/x', which points to '/elsewhere'...
replacing the Git work tree 'host:/var/src/x', because ...
```

A target path in a message starts with `user@host:` for remote targets,
without `user@` if the user is root or not set, and is a plain path for
the local target.  Errors start with `error:` or `krops:`, warnings with
`warning:`.  rsync's itemized changes appear as `%i %n%L %'''b/%'''l`, with sizes in
units of 1024.

## Safety

Populate never changes or removes anything outside the target directory,
whatever the sources and whatever it finds on the target:

- The checks above keep source names and the target path inside it.
- A symlink at the target path of a source gets replaced, never written
  through, and populate's state directory, `.populate`, must not be one.
- Removing a tree makes only its directories writable, and removes what's
  below them, not what symlinks point to.
- Hard links to files of a source get broken before rsync writes to them.
- A Git work tree gets reused only if its `.git` is a directory of its
  own, with no other configuration than `git clone` and populate set,
  and Git runs without hooks and file system monitors.
- Temporary files get created with `mktemp`, and moved into place.

The tests' canaries, files around the target directories and in other
places populate must not touch, check this after every phase.

## Execution

Whether a target is local is what `--local` or `--remote` says, or else
gets decided when the program runs, like krops' lib's `isLocalTarget`
does: it is, if its user is `LOGNAME`, or null if that's empty or unset,
and its host is `localhost` or the machine's name, from `HOSTNAME`, or, if
that's empty or unset, the only line of `/etc/hostname` that is a label.
For a local target, populate runs commands directly.

For a remote target, populate runs commands over ssh, sharing one master
connection per target.  The commands are fixed POSIX shell snippets, with
values passed as positional arguments, so nothing gets quoted into them.
The target needs a POSIX shell, coreutils, findutils, rsync, `patch` for
patches, Git for `git` sources, and Nix; NixOS lacks Git.  With `sudo`,
the snippets run via `sudo`.

Populate runs rsync, Git, ssh, and Nix as programs, found in `PATH`.

## Development

- `cargo test` in this directory runs the program's unit tests.
- `nix-build tests -A smoke` runs the smoke test, a few scenarios covering
  each source type and each kind of target.
- `nix-build tests` runs all scenarios but the exhaustive ones.

## Status

The program delivers all source types, draws inputs from sources, makes
backups, and prunes.  It decides what to do itself: the snippets it runs
on the target send back what it needs to know, e.g. what Nix prints as
JSON, where paths resolve to, or store paths.  Delivering flake and
flake-input sources is in `src/flake_input.rs`, drawing inputs and
deriving copies in `src/flake.rs`.