The main issue is the handling of security updates within the Nixpkgs ecosystem, which relies on Nix’s CI system, Hydra, to test and build packages. Due to the extensive number of packages in the Nixpkgs repository, the process can be slow, causing delays in the release of updates. As an example, the updated xz 5.4.6 package took nearly 5 days to become available in the unstable branch!
Fundamentally, there needs to be a change in how security fixes are handled in Hydra. As stated in the article, Nix was lucky to be unaffected, but multiple days to push out a security patch of this severity is concerning, even if there was no reason for concern.
Kinda tired of the constant flow of endless “analysis” of xz at this point.
There’s no real good solution to “upstream gets owned by evil nation state maintainer” - especially when they run it in multi-year op.
It simply doesn’t matter what downstream does if the upstream build systems get owned without anyone noticing. We’re fucked.
Debian’s build chroots were running Sid - so they stopped it all. They analyzed and there was some work done with reproducible builds (which is a good idea for distro maintainers). Pushing out security updates when you don’t trust your build system is silly. Yeah, fast security updates are nice, but it took multiple days to reverse the exploit, this wasn’t easy.
Bottom line, don’t run bleeding edge distros in prod.
We got very lucky with xz. We might not be as lucky with the next one (or the ones in the past).
I think the post was more about pointing out how long it takes to put out a security patch. Security patches can also occur on stable.
I’m not sure why you think I didn’t? Sorry if it was unclear.
From the blog:
This incident has really made me wonder if running the unstable branch is a great idea or not.
My comment:
Bottom line, don’t run bleeding edge distros in prod.
Hope this clarified my opinion! Have a good day!
Were systems in the stable branch at risk of compromise? Were there delays in releasing security fixes in the stable branch.
I don’t even think unstable was suseptical to it. I don’t think Nix ties ssh to systemd. Debian and redhat do.
AFAIK it was enabled in anything that used the official source tarball. The exploit binaries were added during the tarball build process.
This blog post misses entirely that this has nothing to do with the unstable channel. It just happened to only affect unstable this time because it gets updates first. If we had found out about the xz backdoor two months later (totally possible; we were really lucky this time), this would have affected a stable channel in exactly the same way. (It’d be slightly worse actually because that’d be a potentially breaking change too but I digress.)
I see two way to “fix” this:
- Throw a shitton of money at builders. I could see this getting staging-next rebuild times down to just 1-2 days which I’d say is almost acceptable. This could even be a temporary thing to reduce cost; quickly renting an extremely large on-demand fleet from some cloud provider for a day whenever a critical world rebuild needs to be done which shouldn’t be too often.
- Implement pure grafting for important security patches through a second overlay-like mechanism.
Isn’t that the risk of running an unstable build of anything?
-
You’re using the unstable channel.
-
You could literally build it on your own, or patch your own change without having to wait - all you have to do is update the SHA256 hash and the tag/commit hash.
-
You can use slightly older official channels, as well as patched unofficial channels.
-
You can go back to using older binary, while running GC on suspected, newer malware binary, assuming that it hasn’t been executed yet.
If you’re not using Nix the way it is intended to be, it is on you. Your over-reliance on Hydra is not the fault of Nix in any way.
First of all, I’m not the author of the article, so you’re barking up the wrong tree.
You’re using the unstable channel.
That doesn’t matter in the big scheme of things - it doesn’t solve the fundamental issue of slow security updates.
You could literally build it on your own, or patch your own change without having to wait - all you have to do is update the SHA256 hash and the tag/commit hash.
Do you seriously expect people to do that every time there’s a security update? Especially considering how large the ecosystem is? And what if someone wasn’t aware of the issue, do you really expect people to be across every single vulnerability across the hundreds or thousands of OSS projects that may be tied to the packages you’ve got on your machine?
The rest of your points also assume that the older packages don’t have a vulnerability. The point of this post isn’t really about the xz backdoor, but to highlight the issue of slow security updates.
If you’re not using Nix the way it is intended to be, it is on you. Your over-reliance on Hydra is not the fault of Nix in any way.
Citation needed. I’ve never seen the Nix developers state that in any official capacity.
After thinking a lot, your stance does make sense. However, it is not for the points you’ve raised in your defense - they’re not strong reasons that support Nix’s flaw, if we are talking about a generic trojan. The store-based file hierarchy would serve as a sort of defense, however, it isn’t a fool proof security.
However, if its solely targeted keeping Nix in mind, then yes, it’s a pile of stinking mess, with how the entire nixpkg file is filled with diff files. Diff files are not only difficult to read, it is also very easy to inject code without anyone finding out. It’s a ticking bomb waiting to explode.
