aboutsummaryrefslogtreecommitdiffstats
path: root/pkgs/populate/docs/design.md
blob: 44d8b21c68a376d2badd258bfe15c58e42b9df4e (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
# Populate's design

Populate delivers sources to a target directory.  It consists of two parts:

- Nix code, which turns populate's arguments into a spec: the sources as
  `evalSource` returns them, the target, and the options, as JSON.  It
  checks the spec during evaluation, so mistakes show early.
- A program, which reads a spec and delivers the sources.  It knows
  nothing about Nix expressions: whatever produces a spec can drive it.

So there is one program for each version of populate, instead of a script
for each set of arguments, and the program can be written in any language.
This directory holds the program in Rust.  Another implementation, e.g. in
Haskell, reads the same spec, prints the same messages, and passes the same
tests.

## Nix side

`default.nix` is `pkgs.populate`.  Given populate's arguments, `{ backup ?
false, force ? false, prune ? false, source, target }`, it checks them with
`checks.nix`, turns them into a spec, and returns a script running the
program with the spec on file descriptor 3, so the commands populate runs
keep standard input.  The script puts the tools populate runs in front of
`PATH`, and depends on the store paths in the spec, so building it builds
them.  `rustProgram` is the program, by default built by `program.nix`.

`writeDeploy` and the other functions of `pkgs.krops` use `pkgs.populate`,
so their arguments stay as they are.

## Invocation

```
populate SPEC
```

`SPEC` is the path of a file containing the spec, or `-` for standard
input.  Exit status:

- 0: populate delivered all sources.
- 1: a check of the spec failed, or delivering did.
- 2: the arguments are wrong, or the spec can't be read, isn't JSON, is of
  another version, or doesn't have the spec's shape.
- 129, 130, 143: populate got SIGHUP, SIGINT, or SIGTERM, and stopped.

## Spec, version 1

The spec is a JSON object:

| Field     | Type                 | Default | Meaning                                       |
|-----------|----------------------|---------|-----------------------------------------------|
| `version` | number               |         | The version of the spec: 1.                   |
| `backup`  | boolean              | false   | Copy the sources to backup first.             |
| `force`   | boolean              | false   | Create the sentinel if it's missing.          |
| `prune`   | boolean              | false   | Remove what isn't a source.                   |
| `source`  | object of sources    |         | The sources, by name.                         |
| `target`  | object               |         | Where to deliver them.                        |

`target` has these fields:

| Field          | Type             | Default | Meaning                                  |
|----------------|------------------|---------|------------------------------------------|
| `user`         | string or null   | null    | The user to log in as.                   |
| `host`         | string           |         | The host to deliver to.                  |
| `port`         | string or null   | null    | The port ssh connects to.                |
| `path`         | string           |         | The target directory.                    |
| `sudo`         | boolean          | false   | Deliver as root, via sudo.               |
| `extraOptions` | array of strings | []      | More arguments for ssh.                  |
| `local`        | boolean or null  | null    | Whether the target is local.             |

Each source is an object with the field `type`, the source's type, and a
field named after the type, holding the source's settings:

```json
{ "type": "git", "git": { "url": "https://...", "ref": "main", ... } }
```

The types and their settings are those of `lib/types/populate.nix`, as
`evalSource` returns them: with shorthands expanded, e.g. `file =
"/etc/x"` as `{ "path": "/etc/x" }`, and without settings that are null.
The source types are `derivation`, `file`, `flake`, `flake-input`, `git`,
`pass`, `passage`, `pipe`, and `symlink`.

Fields the spec doesn't define are errors, so a misspelt option doesn't go
unnoticed, and a spec of another version gets refused before anything
else.  A new version of the spec adds fields or changes their meaning; the
program reads the versions it knows.

Store paths in the spec, e.g. of a `file` source that is a derivation, must
exist when the program runs.  The script `default.nix` writes depends on them.
Something producing a spec at run time has to build them first.

## Checks

The program checks the spec as `checks.nix` does, with the same messages:

- Source names are portable file names: letters, digits, and `._-`, not
  starting with `-`, and neither `.`, `..`, nor `.populate`.
- The target path is absolute, not the root directory, and contains
  neither `..` nor control characters.
- Inputs get drawn from sources that exist, not from the source itself,
  and not in circles.

Checks that depend on whether the target is local come on top, e.g.
refusing to copy flakes to remote targets using sudo.  `checks.nix` does
that during evaluation too, where `default.nix` decides whether the target
is local.

## Output

What populate prints on standard error is part of its interface: people
read it, and the tests check it.  For example:

```
populating 'host:/var/src/nixpkgs'...
pruning 'host:/var/src/old'...
replacing symlink 'host:/var/src/x', which points to '/elsewhere'...
replacing the Git work tree 'host:/var/src/x', because ...
```

A target path in a message starts with `user@host:` for remote targets,
without `user@` if the user is root or not set, and is a plain path for
the local target.  Errors start with `error:` or `krops:`, warnings with
`warning:`.  rsync's itemized changes appear as `%i %n%L %b/%l`.

## Safety

Populate never changes or removes anything outside the target directory,
whatever the spec and whatever it finds on the target:

- The checks above keep source names and the target path inside it.
- A symlink at the target path of a source gets replaced, never written
  through, and populate's state directory, `.populate`, must not be one.
- Removing a tree makes only its directories writable, and removes what's
  below them, not what symlinks point to.
- Hard links to files of a source get broken before rsync writes to them.
- A Git work tree gets reused only if its `.git` is a directory of its
  own, with no other configuration than `git clone` and populate set,
  and Git runs without hooks and file system monitors.
- Temporary files get created with `mktemp`, and moved into place.

The tests' canaries, files around the target directories and in other
places populate must not touch, check this after every phase.

## Execution

Whether a target is local is what the target's `local` says, or, if
that's null, gets decided when the program runs: it is, if its user is
`LOGNAME` and its host is `localhost` or the host's name, from `HOSTNAME`
or `/etc/hostname`.  `default.nix` decides during evaluation, like the other
functions of krops.  For a local target, populate runs commands directly.

For a remote target, populate runs commands over ssh, sharing one master
connection per target.  The commands are fixed POSIX shell snippets, with
values passed as positional arguments, so nothing gets quoted into them.
The target needs a POSIX shell, coreutils, findutils, rsync, Git for
`git` sources, and Nix for flakes; fresh installations of NixOS lack rsync
and Git.  With `sudo`, the snippets run via `sudo`.

Populate runs rsync, Git, ssh, and Nix as programs, found in `PATH`.

## Development

- `cargo test` in this directory runs the program's unit tests.
- `nix-build tests -A smoke` runs the smoke test, a few scenarios covering
  each source type and each kind of target.
- `nix-build tests` runs all scenarios but the exhaustive ones.

## Status

The program delivers all source types, draws inputs from sources, makes
backups, and prunes.  It decides what to do itself: the snippets it runs
on the target send back what it needs to know, e.g. what Nix prints as
JSON, where paths resolve to, or store paths.  Delivering flake and
flake-input sources is in `src/flake_input.rs`, drawing inputs and
deriving copies in `src/flake.rs`.