Debian on xz-utils: revert to version that does not contain changes by bad actor
bugs.debian.org
bugs.debian.org
(Experts: how do you weigh the risks of the two choices against each other? Running bleeding-edge upgrades brings in these fresh security holes before anyone's noticed them. But, running "stable" software can mean running vulnerable software that upstream has already patched. Not every vulnerability is recognized as a vulnerability, and patched as a security update; many "ordinary" bugs have security consequences which aren't obvious to recognize. Which choice is less worse?)
On the whole, I would assume that someone who maintains a Debian or RHEL package is better qualified to assess the severity of a bug than I am. The bigger problem is that xz hasn't had a proper maintainer for the better part of a decade.
Is this common? Is it more common in software that has lower profile than the Linux kernel—software with fewer maintainers and fewer eyeballs, and fewer security-focused people flagging the right issues in the right ways to downstream?
the reason $currentDumpsterFire is identified is only because it's being used. Cannon fodder exists for a purpose.
Stable is LTS, sid is cutting edge, and testing is somewhere in between. You can choose the level of stability you want, and even move from one to the other, all without leaving the umbrella of Debian.
Well, it doesn't ship systemd whatsoever, but that's a plus to me.
Edit: A Russian sounding alias was also used.
Or maybe they actually are Chinese, and want me to believe the sentence above. :)
Spreading doubt is taught in bad actor-academy.
I think your comment is facetious and adds nothing to the conversation.
Neither this one, nor the previous one.
Jie would be female, Jia male
While gendered names do exist, you can't easily tell from the romanization, it depends on the exact characters used. And most likely "Jia" is the surname, or rather the romanization of one of five different surnames.
Also from that writeup. It also links to https://rheaeve.substack.com/p/xz-backdoor-times-damned-time... which suggests the actual TZ of 'Jia Tan' to be UTC+02/03
We can argue about the specific implementation, but to suggest that it’s impossible seems incorrect to me.
Haven’t nation state actors openly infiltrated high level companies? This would provide a false sense of security imo. If anything, we need better testing and behavior heuristics for incoming code.
I develop cool thing X. Now, 100 minor things depend upon it.
Suddenly, Facebook (or anyone of that size!) starts using it, and decides to vet the maintainer/author.
Who says anyone has to cooperate? It's his software. He wrote it. Don't like it?
Well tough!
Now obviously Facebook could author a replacement. It could fork and maintain.
But the very nerve that Facebook(or anyone!) would insist upon a security audit of the anonymous author would be very, very strange.
Next up, I lend a neighbour my lawn mower, after he comes begging to borrow it. Oh but wait! My neighbour now wants me to sign a libabilty form, and also undergo a security check, all so he can borrow my lawnmower!
The hell?!?!
Hoping this illustrates my point. The project author owes nothing to anyone.
And it gets more wacky, if there are 100 companies demanding audits. What? Demand?!
This is where distros are the strong point. They aren't perfect, but they catch a lot of stuff on their own. And maintainers of different distros often backchannel, support each other in this.
In terms of some government org "vetting" people? Way to take the last vestiges of free software, and hacking, and turn it into a gatekeeping, bureaucratic nightmare. I guess one will need credentials, government id, a 10 year security check, to be fingerprinted, and so on? Security clearances work like that, and that's how you vet someone.
I’m also personally torn on how much we want giant private companies controlling more and more of the core compute infrastructure and software.
In an ideal world, the code and software itself would be automatically analyzed for malicious use cases in release and deployment pipelines, but that’s a magic hand wavy kind of ask of a huge magnitude and complexity.
The point of the anecdote is to supplement the parent poster by stating that their hypothetical scenarios are already happening.
We should have better testing and more eyes on incoming code for projects we depend on. But my point I guess is that vetting maintainers is not an option.
Same reason why commercial planes always have two vetted pilots in the cockpit and ATC watching their every move. People can and will do malicious shit, if their mental health declines.
Fwiw, it’s possible to use zstd instead of xz, which was created at and actively maintained at Facebook.
So if you were expecting Big Tech to identify this class of problem and do something about it, it seems like they are. But if you expected them to eliminate every single possible instance of such an issue, that’s a bit much isn’t it?
Build system is complex because of legacy and need to support millions of different platforms and distros.
Sure; you can hide nasty code in rust. But I think it would have been harder. Not because rust is memory safe. But because rust is decades newer, and it doesn't inherit all of the quirks in C, in its build system and obscure mechanisms for dynamic linking.
Bingo. This was a highly motivated, targeted, and well planned compromise of the supply chain itself. And funnily enough, using a memory-safe language could have actually made it less likely for this to be detected.
There were symptoms showing up prior to the discovery, most prominently Valgrind errors in otherwise unrelated project builds. (ref: https://bugzilla.redhat.com/show_bug.cgi?id=2267598)
As a result, there is a sentiment in the peanut gallery that "memory safe language == Rust".
You are correct in that a segfault in a project written in memory safe language gets more scrutiny. But imagine for a moment what would have happened if the compiler guaranteed that those Valgrind-caught memory access violations could not have occurred in the first place?
We got lucky, and the developers who wrote the actual backdoor were sloppy. They were aiming for sneakiness and evasion first, memory correctness would have likely been an afterthought. The next group tasked with a similar approach will know better.
But it’s a moot point because there are undoubtedly other mechanisms a malicious maintainer could use to inject a vulnerability. Just not this exact one.
The underhanded C contest would be a lot less entertaining in rust. Thats a good thing.
It wouldn't solve the underlying problem permanently. But I don't think there's any slam-dunk solution here. I'll take any small wins where we can get them.
>The compiler supports various methods to link crates together both statically and dynamically. This section will explore the various methods to link crates together, and more information about native libraries can be found in the FFI section of the book.
I’m talking about what happens in practice, which is shipping static binaries that contain all of their dependencies, except for libc.
Introducing such hidden bugs that are basically impossible to find is much harder in a memory safe, statically linked language.
Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
tar -xf lzip[version].tar.gz
cd lzip[version] && ./configure && make check
Yeah, this is going to massively improve things. I foresee no plausible way we can gave a repeat if we switch to this lzip thing. Absolutely watertight. We learned from past events. Problem solved.
The backdoor was in the released tar only, the sources were fine, because such a malicious m4 would have been detected. The compressed test bin containing the backdoor was in git, but such unobvious things do get attention.
By the xz maintainer? I very much doubt that.
Absolutely false because Zstandard didn't exist when xz was initially chosen. And the actual ratio difference is not that huge, generally less than 10% for most use cases, while the fastest level of xz is slower than most levels of zstd. (In fact, `dpkg-deb` uses `xz -6` by default because `-9` would be really expensive.) Given that zstd is also faster to decompress than xz as well, the ~10% increase is justifiable.
> Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
Lzip is not a better-designed format, it is just needlessly minimal for its use cases, much like xz's needless extensibility. I have detailed my own criticisms in the other comment [1].
"Is xz-utils going to be maintained? Will we want to keep in the archive an unmaintained low-level library - low-level as in, susceptible of getting pulled as a dependency in lots of places - and rely on it for components such as dpkg?"
Ref XKCD 2347.
[1] https://lists.debian.org/debian-devel-announce/2011/08/msg00...
If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.
In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...
If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario.
A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.
While matching LZMA2 compression ratio: no.
The backdoor was added to initialization code in liblzma (which runs whenever the dynamic linker loads that library). So that malicious code runs before sshd starts executing, and manipulates the dynamic linker so that it will do something (which is still being analyzed AFAIK) when some specific functions are called by the sshd executable.
So it is wrong for such a fundamental service as systemd to depend on such complicated processes ... just as the anti-systemd people said. Except of course shellscripts and /bin programs are just as complicated, when you check.
Also, I am a strong believer of strength by diversity, especially for anything that is safety-critical.
[1] Except when there are C/C++ dependencies :-)
It doesn't seem wise to accept contributions to projects which may have security implications from anons. Code contributions clearly can't or aren't independently validated.
I also recommend this summary [1].
[0] https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h... [1] https://boehs.org/node/everything-i-know-about-the-xz-backdo...
Taking contributions from anons appears to be common. I am suggesting that should change.
EDIT: To be clear, this problem is not directed to the original xz maintainer, but more about how to prevent or reduce such appointments in the first place.
https://www.bankid.com/en/foretag/secure-digital-identificat...
I think India has something similar?
Not sure if open source developers would be so keen on introducing requirements like that into the development process. Some developers want to keep a low profile and not reveal to much about their AFK identity. And nearly all tech for doing stuff like this is proprietary.
GitHub could provide this as a service, perhaps using uploads of whatever state/national ID the user is coming from. Give a little 'verified ID' badge. All banks do things like this for new accounts.
Granted this doesn't stop state-actors, but it should be effective against cyber criminals.
I think a system where it was harder for unknowns/newcomers to contribute but had better security is a good tradeoff. Esp. since it is likely this incident is not unique.
How are you going to verify a national ID? By taking a photo of it? What if the photo is edited? By somehow going to the maintainer in person and looking for anti-counterfeiting features in the paper? What if the ID is old enough that it predates these anti-counterfeiting features? Not all countries are Germany.
> or verification by an employer.
What if the maintainer is unemployed? What if the reason the maintainer has enough free time to maintain the package is because they're already retired, and living off their pension or investment returns?
> Both are ubiquitous
From what I've heard in the Internet, some large countries still don't have a national ID.
How many actually existing maintainers in particular of security sensitive parts of open source projects don't have at least some record of employment or education, a modern passport or national ID with eID features, some other comparable form of identification or at least say one reputable person in the community who can verify they are who they say they are?
Are we seriously pretending the standards that exist for ordering a beer in a bar can't be met for software security that literally billions of people depend on?
So just claim that you are verified by your employer? It's fairly easy to setup a web page for a non-existing company...
Consider also that in many countries the government can force a honest national to collaborate with them on introducing backdoors to any target. This can happen both in OSS and proprietary software projects btw
Every project potentially has security implications. A package that's used for compression today could be linked to an SSH server tomorrow. Something as banal as a todo list could be a potential attack vector in the right hands.
I don't know what the solution to this problem is. Better verification of dependencies, certainly, but how? More investment by corporate users of FOSS into security? But no matter what we do, "who's watching the watchmen" will remain a valid question.
I'm actually quite impressed that this was found as quickly as it has, though it makes me wary that there might be other, as yet unfound, such attacks.
Has it been shown this is by a state-level actor?
Why? Still throwing out the baby with the bathwater. Is there a huge number of problematic open source commits by "cyber criminals"? Would this even stop them? Doubtful. Will this make it harder for bona fide people to contribute? Certainly.
I feel it is not really worth it to vet 2 years' worth of changes by a malicious actor, who had been actively trying to make any exploits add by them as hard to discover as possible, when good alternatives, like zstd, exist.
What we don't know for sure is whether Jia Tan (whoever that might be, if it is even a single person) did it on purpose, or if they had their account compromised.
Next, the affected codepath skips itself if it's not executed under just-so conditions. Specifically testing to see if it's being run interactively to avoid diagnosis, and testing to see if the executed binary is sshd.
Lastly, what good reason could a compression library possibly have, for overriding encryption routines in sshd?
It's a fair question, but it doesn't survive inspection at all. This is like finding someone in a ninja suit, wearing night vision equipment, creeping around a museum in the middle of the night, with the crown jewels in a bag slung over their shoulder - and asking yourself if they're perhaps a lost tourist.
The payload already is pretty sophisticated and full of intent to both hide itself and to target sshd very specifically. And on top, there is another layer of pretty clever build infrastructure and manipulation - again with fairly clear intent to hide the inclusion of the payload. Not to take away from the discovery, but finding this was lucky.
But then you have two actions with imo fairly clear intent to hide, deceive and target security infrastructure - and then some more mundane actions become weird as well. Deactivating parts in oss-fuzz is plausible under the right conditions, but in this light? Or being pushy to get this released into newer Fedora versions - again, might be understandable and boring in other circumstances.
Staying in your example - suddenly it becomes strange that this person got lost in the museum two or three times over the last month. Or has been taking walks around the museum very often over the last weeks. Nothing illegal with those on their own, but like that?
Cumulus Linux (originally by Cumulus Networks, which is now part of Nvidia) is a Debian-based Linux distribution used the data plane in datacenters: Dell, HPE, Mellanox, Lenovo and others let customers use Cumulus Linux. That means major datacenters running a potentially-compromised Linux distribution because of the xz attack.
McAfee Linux used to be (it EOL'ed recently) a CentOS-based distribution (which in turn comes from Fedora) used for security appliances by McAfee. Back when the xz attack was initiated (Oct'21), McAfee Linux was in full support. Even more enterprise-grade appliances to be easily compromised.
As a former Debian developer, I wonder if the "must use upstream tarball" approach Debian has (or at least, when I was a DD, just a handful of years ago) makes sense. It seems if built from the git repository, this xz attack would not have succeeded.
PS: why the negatives to an honest question!???
It was really bad.
AFAIK there is no evidence that CVE-2008-0166 was done deliberately, whereas with the xz backdoor, there seems to be lots of signs of deliberate intent.
Debian is also 100% at fault here and I hope this slap in the face makes them reconsider their poor security practices. Amateur package maintainers should not be overstepping their job (packaging) and enabling disastrous vulnerabilities in security-critical software.
The creation of this backdoor did not take place in a vacuum. Can you imagine the same person trying to introduce build changes and binary test cases into openssh? This backdoor was crafted in this particular way _specifically_ to exploit the fact that sshd in distributions such as Debian is being indirectly linked with lzma.
At least there is no lack of amateur enthusiasts telling people how they are wrong. :)