aboutsummaryrefslogtreecommitdiffstats
path: root/ci.nix
diff options
context:
space:
mode:
authortv <tv@krebsco.de>2026-11-12 02:07:10 +0000
committertv <tv@krebsco.de>2026-11-12 02:07:10 +0000
commit1c7621261c4e9df8d572aef5eb6369d20eea23aa (patch)
tree1ffd240897052146a2c1cebe54b699b052dbd237 /ci.nix
parent3c062a788580a846acb16a5a9ad519bbee2c3747 (diff)
tests: isolate flake scenarios, check eval errors against controls
Several weaknesses let checks pass although the feature under test is broken: - Nix restores fetched inputs from its fetcher cache, so evaluating a delivered flake offline after collecting garbage succeeds even if populate rooted none of its inputs. nix_eval uses an empty cache directory. - Scenarios share target directories and stores, so store paths and roots of earlier scenarios stand in for missing ones, and uploads of flakes never run in a passing scenario. Scenarios of the flake, flake-input, and features tests are isolated now: they start without target directories and with a collected store on both machines. - Phases expecting evaluation to fail pass on any evaluation failure, e.g. a misspelt option. Each such phase has a control now: sources differing only in what makes evaluation fail, which must evaluate. Further, populate runs with a time limit, so a hang fails its scenario instead of the whole test, phases can run steps on the deploying machine before and after, and evaluating the scenarios fails if the remote target counts as local for whoever evaluates them.
Diffstat (limited to 'ci.nix')
0 files changed, 0 insertions, 0 deletions