nixpkgs-review: Determine regressions from target branch
I would be more than okay with building the target branch’s packages to see if any regression happened as part of a nixpkgs-review. Something to the affect of New successes, new failures, still failing would be nice. Similar to hydra evaluations.
My current workflow is to take all the packages that failed and run nix build -f . --keep-going $@. And then manually see which ones passed. This isn’t too bad if it’s just <10 failures, but some reviews (especially python packages), can have 20-60 failures, and it becomes very difficult to determine regressions.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 6
- Comments: 16 (15 by maintainers)
--regressionsseem to be the best most expressive one becausenix-reviewis already testing changes, but not the packages before they were changed.This only works for local evaluation right now, though.
--deltawould also work, but it’s a little “math-y”.--package-diff--set-diffmaybe
--changes?--regressions?actually, maybe it would be good to take a break from constant PR reviews
Yes. It is just an optimization, but there is no need to actually download it or to rebuild an already failed build.
Also when targeting master, you’re likely to get cached hits if they weren’t broken already
Yea, but I don’t really mind brute forcing with more CPU. Would you be willing to work with me if I created a PR that did create the package diffs?
Yes, essentially a “package diff” which told you what changed.