Xzbot: Notes, honeypot, and exploit demo for the xz backdoor
github.com
github.com
If this hadn’t had the performance impact that brought it quickly to the surface, it’s possible that this would have lived quietly for a long time exactly because it’s not widely exploitable.
But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event.
Now it looks more like nation-state spycraft.
Why not? It's possible someone else could've discovered the exploit before the big attack but decided not to disclose it. Or that they could've disclosed it and caused a lot of damage the attacker didn't necessarily want. And they easily could've been planning both a long-term way to access a huge swath of machines and also biding their time for a huge heist.
They have no reason to not restrict the backdoor to their personal use. And it probably is spycraft of some sort, and I think more likely than not it's a nation-state, but not necessarily. I could see a talented individual or group wanting to pull this off.
https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...
https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.h...
https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.h...
While this doesn't prove nation-state involvement, it certainly borrows from a wide-ranging playbook of techniques.
If you, as a maintainer, value stability of not only your software but also your own mental health, it is entirely something you can be proud of to resist calls for new features, scope increases, and rapid team additions.
But it's hard to see the whole thing helping, and it is some combination of depressing and infuriating. I hope he's doing ok.
It's also interesting because it exploits the current zeitgeist when it comes to maintainers' responsibilities to their users and communities. Personally, I think it puts too much expectation on maintainers' shoulders, especially when they're working for free.
Personally, I think maintainers are doing favors for users, and if they don't like how the project is progressing or not, then too bad. That's not a popular sentiment, though.
It seems like they were actively looking[0] which open source compression library they can inject with vulnerable code and then exploit and backdoor afterwards.
https://github.com/libarchive/libarchive/pull/1609#issuecomm...
> "In 2021, JiaT75 submitted a pull request to the libarchive repository with the title ‘Added error text to warning when untaring with bsdtar’ which seemed legitimate at first glance. "
Seems “Jia” is related to both
It could just as easily be a lone wolf capable of writing in different styles (really not difficult if one spends enough time online, one gets exposed to all sorts of styles) or even just your run of the mill criminal gang.
In fact, "multiaccount" cheating (where a single person has multiple accounts that interact with one another in a favourable manner, e.g. trading for stuff for reduced fees, or letting themselves be killed to pad stats, or expressing opinions leaving the impression of mass support, etc.) has been present in many MMO games for more than a decade. I recall an online game I was really into back in high school, eRepublik, which simulated real world politics, economics and warfare (in a really dumbed down version of course), and multiacounts were especially prevalent in politics, where candidates for election would often be padded by fake groups, or even entire countries being taken over by a gang of real people with like ten accounts each.
The complexity is nothing new. The only indicator this could be a state actor is the long con aspect of it. The attacker(s) were looking into years between starting and actually being able to exploit.
https://www.mail-archive.com/xz-devel@tukaani.org/msg00556.h...
That was not necessary to gain trust. Writing style is different, too. Later when Jia gained commit access he reminds him to merge it.
I think the most convincing case made about the sock puppets is around account creation dates, and also people disappearing after they get what they need. Like Jigar disappearing after Jia becomes maintainer. Or the guy "misoeater19" who creates his debian bug tracker account to say that his work is totally blocked on needing xz 5.6.1 to be in debian unstable.
It's literally a couple hours work.
Theoretically, I could even imagine that public key security motivated by some "mostly benign" actor who fell in love with the question "could I pull this off?" dreading the scenario of their future hacker superpower falling in the wrong hands. It's not entirely unthinkable that an alternative timeline where this was never discovered would only ever see this superbackdoor getting used for harmless pranks, winning bets and the like. In this unlikely scenario, the discovery, through the state actor and crime syndicate activities undoubtedly inspired by the discovery, would paradoxically lead to a less safe computing world than the alternative timeline where the person with the key had free RCE almost everywhere.
That's to say an enterprise router or switch would likely have secured exploits put there by corporate and national security agencies, whereas open source exploits would benefit from the probable deniability.
"It seemed like a good idea at the time" is a pretty common story though. ;)
So, while writing the exploit might be a couple of hours work, actually pulling it off is quite a bit more difficult.
For many projects, that is enough to become the main contributor.
I disagree, plenty of exploits implemented WireGuard-like crypto key access to them.
The tradeoff is that, once you find it, it's very clearly a backdoor. No way you can pretend this was an innocent bug.
There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708
I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz.
I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well.
Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this:
``` __attribute__((__used__,__no_sanitize_address__)) ```
to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script.
With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.
It was reported by an MS engineer who happens to be involved in another OSS project. MS is doing business with the US intelligence community, for example there is the Skype story: First, rumors that NSA offers a lot of money for people who can break Skype's E2E encryption, then MS buys Skype, then MS changes Skype's client to not be E2E encrypted any more and to use MS servers instead of peer to peer, allowing undetectable wiretapping of arbitrary connections.
But it's a quite credible story too that it was just a random discovery. Even if it was the NSA, why would they hide that capability. It doesn't take much to run a script to compare git state with uploaded source tarballs in distros like Debian (Debian has separate tarballs for the source and the source with Debian patches applied).
I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).
Perhaps you’re not familiar with what NSA historically stood for: Never Say Anything.
Let's say agency A has some scanning capability on open source software that detected this backdoor attempt by agency B. If they had gone public, agency B now knows they have this ability. So agency B will adjust their ways the next time and the scanning capability becomes less useful. While if agency A had told Microsoft to "find" this by accident, nobody would know about their scanning capability. And the next attempt by agency B would only try to avoid having the performance impact this first attempt had, probably leaving it visible to agency A.
no this engineer is world-known for being core PostgreSQL, a team with high standards.. unlike that company you mention
(2) they could just say they found it during some routine dependency review or whatever
Second, maybe a routine dependency review is how they _actually_ found it but they don’t want future people like this focusing too much on that otherwise they may try to mitigate, whereas now they may focus on something inane like a 0.5 second increase in sshd’s load time or whatever.
Basically "parallel construction". It's very possible it's what happened.
People love when there's some explanation that doesn't involve randomness, because with randomness looks like we don't have grasp on things.
Google actually had tooling that was detecting it, but he disabled check that would show it.
Google/Microsoft/NSA could just say they detected it with internal tooling and not disclose how exactly. Google and Microsoft would love to have credit.
https://www.cnx-software.com/2021/04/22/phd-students-willful...
If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war".
There are shades of "Setec Astronomy" here.
It was a psyop to increase the scrutiny around OSS components.
Kidding. Mostly...
But given the amount of scrutiny folks are going to start putting into some supply chains... Probably cheaper to execute than most company's annual security awareness budgets cost.
one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin.
[1] https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01
Typically its only state agencies that will fund an operation with uncertain pay off over long periods of time. That type of patience is expensive.
Online criminals are beholden to changing market pressures and short term investment pressures like any other start up.
That said it's a lot of work - 2 years at least. It's an exploit that's so good that you'd have to use incredibly carefully - also because if/when it's discovered it's going to break everywhere.
I've read descriptions about how the NSA (and presumably other such agencies) and they're really careful. The first job is to make sure the target doesn't get confirmation that they are in fact a target. The second is that they always cover their tracks so the target doesn't know they were a target.
Criminals tend to do the first but almost never the second so a tool like this - while I'm sure they would love - isn't worth the amount of work it would take to develop.
Again - I'm not 100% on this but ... 40% ? say 20% criminals, 10% lone wolf?
If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.
So if distinguishing infected machines was their intention they definitely over-egged it.
I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off.
Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.
It's practically a good backdoor then, crypto graphically protected and safe against "re-play" attacks.
You can just treat the entire thing as opaque and proxy everything to the host you're trying to compromise; as soon as you have an exploit string for a given host you can just replay it.
E: see RFC 4253, sections 8 and 10, and RFC 4252 for corroboration
But you're right: we've verified the hostkey by the time we send the special certificate. So there's no way to effectively replay this without access to the server keys. My original comment is incorrect.
I'm actually surprised there's no logging at INFO or higher after this succeeds, given that openssh typically logs preauth connection closes. But I guess the crutch is that we never log connection opens and we only really log failures in handshaking, and it's not like the backdoor is going to go out of its way to log the fact that it opened itself...
If they determine the vulnerability is only exploitable by the NSA for reasons
such as computational resources, budget, or skill set, they label it as NOBUS
and will not move to patch it, but rather leave it open to exploit against current
or future targets.
If (!) the NSA regards ssh keys as secure, then from that article it sounds like the NOBUS thing would fit.If they didn't have the key though, then yeah it doesn't fit. Unless they can walk through ssh key security anyway. :)
I'd like to rule-out some state actors by the mere heartbreaking act of socially engineering and targeting a person in pain.
Not even considering the original maintainer may be a US citizen (idk).
It looks like the whole contribution to xz was an effort to just inject that backdoor. For example the author created the whole test framework where he could hide the malicious payload.
Before he started work on xz, he made contribution to libarchive in BSD which created a vulnerability.
Comment in your second link:
https://github.com/libarchive/libarchive/pull/1609#issuecomm...
(a) not exploitable without the existence of a much higher-severity exploit -- sure you can clear the screen, but that's low impact
(b) possible to trigger on other extant paths; see https://github.com/libarchive/libarchive/issues/2107 so it seems nothing new was introduced
(c) kind of contrived to get to execute; you have to somehow fail to extract the archive but on the happy path you'll see a bunch of weirdly-named files
I think it's distinctly higher-probability that this change was just meant to build credibility for the GitHub account. The diff is a fairly trivial patch to a minor issue filed around the same time as the pull request.
It's obvious, basically no one knows what's going on in the _vast_ majority of code running out systems these days. And even if you know 99% the attackers only need to be right once
"It's not only the good guys who have guns."
Computers don't know who's / what's good or bad. They're deterministic machines which responds to commands.
I don't know whether there'll be any clues about who did this, but this will be the poster child of "you can't have backdoors and security in a single system" argument (which I strongly support).
I find most interesting how they chose their attack vector of using BAD test data, it makes the rest of the steps incredibly easier when you have a good archive, manipulate it in a structured method (this should show on a graph of the binary pattern btw for future reference) then use it for a fuzzy bad data test. It’s great.
The rest of the techniques are banal enough except the most brilliant move seems to be that they could add “patches” or even whole new back doors using the same pattern on a different test file. Without being noticed.
Really really interesting, GitHub shouldn’t have hidden and removed the repo though. It’s not helpful at all to work through this whole drama.
Edit: I don’t mean to say this is banal in any way, but once the payload was decided and achieved through a super clever idea, the rest was just great obfuscation.
If I was letting someone maintain my codebase, I would 1 Billion percent be reviewing everything… if there was a binary added, I’d be building it myself and comparing the checksums.
Trust absolutely no-one. If you can give a close friend or a loved one a loan of money and it is so easy for them to never pay you back, it should be a reminder that devs you’ve never met are even more likely to scam you.
I think the OpenBSD developers are right to be as paranoid as they are. Anyone who is maintaining a security critical system should be on guard against these kinds of attacks.
But whose problem is that? systemd chose to link against liblzma, not the other way around. I doubt the xz maintainer(s) make money off of the project, and I'm assuming it's a spare-time/side-project type thing. Why should the fact that the library is included in every distro and is a dependency of systemd affect the xz maintainers' obligations? The leader of systemd has been variously employed by RedHat and Microsoft... if they're choosing to pull in an external dependency for systemd and then making money off of selling their Linux distros/cloud services, it would seem they're the ones that could afford to take on the burden of reviewing everything with a fine-toothed comb, not the xz maintaners.
These long-games are kinda scary and until whatever “evil” is actually done you have no idea what is actually malicious or just weird.
Have you considered reaching out to the maintainers of that project and (politely) asking them to explain? In lieu of recent events I don't think anyone would blame you, in fact you might even suggest they explain such an oddly specific side effect in a README or such.
That's kind of a catch-22, right? They'd explain with a seemingly good answer if they did it for actual reasons. They'll still explain with a seemingly good answer if they did it for nefarious reasons.
I don't have a good answer to this except to monitor this dependency and its changes.
The crazy thing is the devs responsible for the XZ backdoor are among us and will likely be here on HN, downvoting comments which are getting close to the truth and upvoting each other.
The bad people are definitely here.
You assume they'll have a good reason, but they may not. They could be a malicious actor but also suck at it, and you might spook them into reconsidering. Or they may be fully innocent and just suck at packaging and distributing software and you might help them learn something.
If they do have a good reason, it should stand on its own merits when presented. Anyone could be up to no good and they might be plotting the end of the world, but they also might coincidentally and not relatedly happen to like xz-utils a lot.
Edit: I think what's next for anyone is to figure out a way to probe for vulnerable deployments (which seems non-trivial) and also perhaps possibly ?upstreaming? a way to monitor if someone actively probes ssh servers with the hardcoded key.
Kudos!
A way to check if servers are vulnerable is probably by querying the package manager for the installed version of xz. Not very sophisticated, but it'll work.
To understand the exact behavior and extend of the backdoor, this does matter. An end to end proof of how it works is exactly what was needed.
> A way to check if servers are vulnerable is probably by querying the package manager
Yes, this has been know since the initial report + later discovering what exact strings are present for the payload.
https://github.com/Neo23x0/signature-base/blob/master/yara/b...
> Not very sophisticated, but it'll work.
Unfortunately, we live in a world with closed-servers and appliances - being able as a customer or pen tester rule out certain class of security issues without having the source/insights available is usually desirable.
Yeah but these servers and appliances aren't running Debian unstable are they? I'd understand if it affected LTS versions of distros, but these were people living on the bleeding edge anyway. Folks managing such servers are going to be fine running `apt-get update`.
We got lucky with this one, tbh.
You misunderstand me, the "need to check for vulnerable servers" has nothing to do with the PoC in itself. You want to know whether you're vulnerable against this mysterious unknown attacker that went through the all the hoops for a sophisticated supply chain attack. I never said that we need a way to detect it because there is a POC out, at least I didn't meant to imply that either.
> script kiddies to use it, meaning we're in a race against them
This is something you and the other person were suddenly coming up with, never said this in first place.
The best one could do is more micro-benchmarking, but for an arbitrary Internet host you aren't going to know whether it's slow because it's vulnerable, or because it's far away, or because the computer's slow in general -- you don't have access to how long connection attempts to that host took historically. (And of course, there are also routing fluctuations.)
The blog has the actual rule content which I haven't seen from other security vendors
https://sysdig.com/blog/cve-2024-3094-detecting-the-sshd-bac...
I don’t think GuardDuty would catch it as it doesn’t look at processes like an EDR does (CrowdStrike, Carbon black), I don’t think sysdig would catch it as looks at containers and cloud infra. Handwaving some complexity here, as GD and sysdig could prob catch something odd via privileges gained and follow-on efforts by the threat actor via this exploit.
So imo means only EDRs (monitoring processes on endpoints) or software supply chain evaluations (monitoring sec problems in upstream FOSS) are most likely to catch the exploit itself.
Leads into another fairly large security theme interestingly - dev teams can dislike putting EDRs on boxes bc of the hit on compute and UX issues if a containment happens, and can dislike limits policy and limits around FOSS use. So this exploit hits at the heart of a org-driven “vulnerability” that has a lot of logic to stay exposed to or to fix, depending on where you sit. Security industry’s problem set in a nutshell.
The main thing I was thinking is that the audit hooking and especially runtime patching across modules (liblzma5 patching functions in the main sshd code block) seems like the kind of thing a generic behavioral profile could get but especially one driven by the fact that sshd does not do any of that normally.
And, yes, performance and reliability issues are a big problem here. When CarbonBlack takes down production again, you probably end up with a bunch of exclusions which mean an actual attacker might be missed.
> From down in the same mail thread: it looks like the individual who committed the backdoor has made some recent contributions to the kernel as well... Ouch.
https://www.openwall.com/lists/oss-security/2024/03/29/10
The OP is such great analysis, I love reading this kind of stuff!
This style of fake doubt is really not appropriate anywhere.
1) No-one has been is proven to "be" anyone in this case. Reputation is OSS is built upon behaviour only, not identity. "Jia Tan" managed to tip the scales by also being helpful. That identity is 99% likely to be a confection.
2) People can do terrible things when strongly encouraged or worse coerced. Including dissolving identity boundaries.
The first problem can be 'solved' by using real identities and web of trust but that will NEVER fly in OSS for a multitude of technical and social reasons. The second problem will simply never be solved in any context, OSS or otherwise. Bad actors be bad, yo.
Cookie guy claimed that he got stabbed and that the federal police was involved in the case, which kind of hints that the events were connected to much bigger actors than just 4chan. At the time a lot of people thought its just Muse Group that's involved, but maybe it was a (Russian) state actor?
Because before that he claimed that audacity had lots of telemetry/backdoors which were the reason he forked and removed in his first commits. Maybe audacity is backdoored after all?
Have to check the audacity source code now.
How did the exploit do this at runtime?
I know the chain was:
opensshd -> systemd for notifications -> xz included as transient dependency
How did liblzma.so.5.6.1 hook/patch all the way back to openssh_RSA_verify when it was loaded into memory?
How was this part obfuscated/undetected?
In a compression/decompression test suite, a subtly broken allegedly compressed binary blob is not out of place.
This suggests we need to audit information flow during builds - the shipping production binary package should be reproduceably buildable without reading test data or test code.
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...
I wasn't aware that the rogue maintainer was able to commit himself without any PR review (or he snuck it through PR review) rogue steps in the build process as well that went unnoticed so that he could bundle decompressed `xz` streams from test data, that also patched output .so files well enough to add hooking code to them.
How many "process failures" are described in that process that exist in every OSS repo with volunteer unknown untrusted maintainers?
> volunteer That's the majority of OSS. Only a handful of the projects we use today as a part of the core set of systems in the OSS world actually have corporate sponsorship by virtue of maintainers/contributors on the payroll. > unknown The actor built up a positive reputation by assisting with maintaining the repo at a time when the lead dev was unable to take an active role. In this sense, although we did not have some kind of full chain of authentication that "Jia Tan" was a real human that existed, that's about as good as it gets, and there's plenty of real world examples of espionage in both the open and closed source software world that can tell us that identity verification may not have prevented anything. > untrusted The actor gained trust. The barrier to gaining trust may have been low due to the mental health of the lead maintainer, but trust was earned and received. The lead maintainer communicated to distros that they should be added.
That's the rub here. It's _really easy_ to say this is a process problem. It's not. This was a social engineering attack first and foremost before anything else. It unlocked the way forward for the threat actor to take many actions unilaterally.
What process failures actually happened here? What changes in process do you think would have stopped this?
The attacker had permissions to create GitHub releases, so they simply added it to the GitHub release tarball.
https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01
These don't have any obvious connection to SSH, so maybe it did things even if there was no connection. This could be important to people who ran the code but never exposed their SSH server to the Internet, which some people seem to be assuming was safe.
Does this mean, had this exploit gone unnoticed, the attacker could have executed arbitrary commands as root without even a single sshd log entry on the compromised host regarding the 'connection'?
If the EDR is capable of intercepting the forks, clone() execves, open(), etc, then you can follow the traces. If it's able to deny certain activity based on rules like modifying /etc/ld.so.preload or download files with curl/wget, it'd have made the attacker's life a bit more difficult.
If the attacker loaded a rootkit, then probably you'd have lost visibility of what the attacker did after that. Also not all the EDRs hook all the functions, or they have bugs, so many times you are not able to follow a trace (without pain/guessing).
This telemetry usually is sent to a remote server, so the attacker could not have deleted it.
Wonder if any anomaly detection would work on that
I’m not aware of any services that a) work like this, or b) would be able to detect this class of attack earlier than last week. If someone does though, please share.
This kind of anomaly detection is possible. Not sure how common it is. I doubt it is common.
An EDR tool would be much better to look for an attacker’s next steps. But if you’re trying to catch a nation state they probably already have a plan for hiding their tracks.
That's actually a good principle to adopt overall.
We should treat this attack like an air plane accident and adopt new rules that mitigate the chances of it being successfully carried out again. We might not be able to vet every single person who contributes, but we should be able to easily separate out noisy test data.
Edit: MIT license provides a pretty broad disclaimer to say it isn’t fit for any purpose implied or otherwise.
Honestly, if I were involved in this, I'd hope that it was, say, the FBI that caught me. I think that'd be the best chance of staying out of the Guantanamo Bay Hilton, laws be damned.
I own you now. Sucks to be you I guess. Or maybe that's not how things work a) limits to what licenses can allow you to do and b) limits to who is bound by the license at all.
In the US, you are innocent until proven guilty.
In the US, you are innocent until proven guilty.
Apologies for having to be obnoxious about it, but it deserves repeating.
If you take a approach of presumed guilty where the accused have to prove their innocence: you have a problem. You'd have a very difficult time proving your innocence after being accused of a crime. Anyone can accuse you of a murder that happened at 2am, and your alibi of being asleep won't cut it. Remember, in this bizarro world, it's presumed you are already guilty. You need to convince the prosector and jury you didn't commit the crime. Even if you had 4K night vision video of you sleeping in your bed, it probably won't persuade anyone because the prosecutor will say you pre-recorded the video to deceive a jury, and since you're presumed guilty, it makes sense for you to manufacture evidence in your defense.
This is why guilty until proven innocent is genuinely stupid.
You also cannot avoid criminal charges for a crime simply by shouting "don't blame me"
I can imagine the case of maybe a "King of the Hill"-type game played on bare hardware, where you're actively trying to hack into and destroy other players' systems. Such a thing might have a license saying "you agree we may wipe your drive after downloading all your data", and that might be acceptable in that specific situation. You knew you were signing up for a risking endeavor that might harm your system. If/when it happens, you'd have a hard time complaining about it doing the thing it advertised that it would do.
Maybe. Get a jury involved and who knows?
But somewhere between those 2 examples is the xz case. There's no way a user of xz could think that it was designed to hack their system, and no amount of licensing can just wave that away.
For a real world analogy, if you go skydiving, and you sign an injury against waiver, and you get hurt out of pure dumb luck and not negligence, good luck suing anyone for that. You jumped out of a plane. What did you think might happen? But if you walk into a McDonald's and fall through the floor into a basement and break your leg, no number of "not responsible for accidents" signs on the walls would keep them from being liable.
Even this is a bad example, since it is just gross negligence and not intentional. A better analogy would be if mcdonalds shoots you.
The point about MIT is that they are saying to the world when publishing “as is” folks. Not claiming I haven’t backdoored it for Uncle Sam.in fact I’m not claiming anything, use at your own risk.
It used to be the law to implicitly do this by weak encryption for exports.
The harm in question is causing the backdoor to be inserted in the first place. Its irrelavent what else you do, like selling it, although that could be a separate crime.
> The point about MIT is that they are saying to the world when publishing “as is” folks. Not claiming I haven’t backdoored it for Uncle Sam.in fact I’m not claiming anything, use at your own risk.
Just because you think that is what those words mean, doesn't mean that is what those words actually mean.
> It used to be the law to implicitly do this by weak encryption for exports.
Not comparable. Even now, the MIT license would probably protect you from any consequenes of using super weak encryption. It would not protect you from the hypothetical you setup. They are very different sutuations.
The 30 remaining gitmo prisoners are all W Bush holdovers that all subsequent administrations forgot.
And in fairness, the whole nature of their secrecy means there's no way to know for sure. It might be just a boogeyman kept around as a useful tool for scaring people into not breaking national security-level laws. I mean, it's not as though I want to go around hacking the planet, but the idea of ending up at a CIA "black site", assuming such things even exist, would be enough to keep me from trying it.
* where were the authors of the code, * where the code is stored, * who is attacked, * where are their servers, * who is attacking, * where are they based, * where did they attack from, * ...
IANAL but it seems very complicated from law perspective (we, humanity, don't have a global law)
Edit2: making a bullet point list is hard
One of those things is going to be fraud: if the maintainer is intentionally installing backdoors into their software and not telling the user, there's going to be some fraud-like statute that they'll be liable for.
Not in the United States. https://constitution.congress.gov/browse/article-1/section-9...
Yes, subsection (b) of the CFAA covers attempts at acts described in subsection (a) of the CFAA. Which specific act under subsection (a) do you claim has been attempted?
> literally all of part a
It's not (1) or (2) as no exfiltration has occurred. It's not (3) as a government system has not been targeted. It's not (4) as "the thing obtained consists only of the use of the computer". It's not (5) as no system has been targeted or damaged (based on the statutory definition of 'damage'). It's not (6) as we have no evidence of the attacker sharing their Ed448 private key. It's not (7) as there has been no act of extortion.
> did someone just trip and fall on their keyboard over a period of months to accidentally carry out sophisticated social engineering to write a carefully hidden backdoor into a package targeted at hijacking widely used operating systems?
I'm not arguing that it was a weird accident. I'm arguing that this clearly malicious act does not, based solely on the information we have today, constitute a violation of the CFAA.
In terms of a plausible business model - I mean that's conspiracy to commit crime, like of course that's illegal? And depending on the state actor and the target system - literally treason if it's a cyber attack as a part of a war? You can put whatever you want into a license or contract but obviously that doesn't let you override the actual law
Other countries such as Israel are playing the long-con with very well engineered, multi-year software backdoors. A much harder game to play.
What are some examples?
The NSA's capabilities back in 2008 were pretty astonishing: "RAGEMASTER" A $30 device that taps a VGA cable and transmits the contents of your screen to the NSA van sitting outside! Crazy stuff. Makes you wonder what they've built in the last 15 years.
What is this in reference to?
"The Maxwell-Mossad team steals spy software PROMIS from the United States, Mossad puts an undetectable trap door in it so Mossad can track the activities of anyone using it, then Maxwell sells it around the world (including back to the U.S. -- with the trap door)."
They are an Israel based company, that sell zero-click RCEs for phones and more. Such malicious software was involved in the murder of journalist Jamal Khashoggi.
Their exploits, developed in-house as well as presumably partially bought on the black market, are some of the most sophisticated exploits found in the wild, e.g. https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i...
> JBIG2 doesn't have scripting capabilities, but when combined with a vulnerability, it does have the ability to emulate circuits of arbitrary logic gates operating on arbitrary memory. So why not just use that to build your own computer architecture and script that!? That's exactly what this exploit does. Using over 70,000 segment commands defining logical bit operations, they define a small computer architecture with features such as registers and a full 64-bit adder and comparator which they use to search memory and perform arithmetic operations. It's not as fast as Javascript, but it's fundamentally computationally equivalent.
Exactly my point. They do not sell backdoors.
Don't get me wrong, still icky, but definitely not a "very well engineered, multi-year software backdoors"
And i mean, i'm genuinely curious. I don't think they would be above doing so if they had the opportunity, i just haven't heard of any instances where they were caught doing so.
RCE is a broad category and includes many things. That said, while technically the liblzma thing is an RCE, supply chain attacks aren't really what most people mean when they talk about an RCE vulnerability.
https://en.wikipedia.org/wiki/Triton_(malware) (https://www.justice.gov/opa/pr/four-russian-government-emplo...)
Basing this stuff on email times (especially replies to emails sent that same day) would be more relevant. However. if this operation was pulled off with the precision and opsec that it seems to have been, I wouldn't be too surprised if whatever group is behind this attack would've sent over + funded a developer somewhere in a time zone of their choice.
No doubt any nation state is able to station someone in Russia/Israel/one of the n-eyes countries.
I doubt OSINT will figure out who is really behind this, but I'm sure governments will, soon enough. Whether or not they report the truth of their findings, and if one should trust the official reporting, is yet another tough question.
See the writeup here: https://til.simonwillison.net/clickhouse/github-explorer
That activity look 100% to be from a paid employee to me. He works in the morning and does git maintenance in the afternoon like clockwork... Between 12 and 18... It might also have been multiple employee.
He didn't work on Christmas or new year eve which is extremely telling...
But note that he comes back next year on the 5th, then disappears for 6th and 7th, and comes back at 8th. 7th is the old-style Eastern Orthodox Christmas, as celebrated in e.g. Russia, Serbia, and Georgia, and 6th is therefore the Christmas Eve. Although in Russia official holidays are the entire week from January, 1 to January, 8 inclusive... still, for this kind of work I wouldn't expect them to stick to the letter of the law there.
It is much more likely that this backdoor would have gone unnoticed for months or years. The access this backdoor provides would be used only once per system, to install other APT (advanced persistent threats), probably layers of them. Use a typical software RAT or rootkit as the first layer. If that is discovered, fallback to the private keys you stole, or the social engineer the company directory you copied. If that fails, rely on the firmware rootkit that only runs if it's timer hasn't been reset in 6 months. Failing that, re-use this backdoor if it's still available.
My bet would be that they were after a crypto exchange(s) where they've already compromised some level of access and want to get deeper into the backend.
>Even if the backdoor had gone into production servers it would have been found fairly quickly if used at some scale.
I agree. Yes it's possible the backdoor could've gone unnoticed for months/years but I think the perp would've had to assume not.
The best most paranoid thing is to have multiple networking layers hoping they have to exploit multiple things at once, and whitelist only networking for exactly what you are expecting to happen. (which is completely incompatible with the idea of ssl, we would need a new type of firewall that sits between ALL applications before encryption, like a firewall between applications and the crypto library itself, which breaks a bunch of other things people want to do)
Then once you do all that, of course, you have one particular version that has been audited and can be used as a dependency. Which means that "ship early, ship often" and "move fast and break things" are completely off the table.
In short, it means doing more or less the exact opposite of the trends in the industry for the past 50 years or so. Starting with rewriting everything that we already have.
And it will be insanely expensive, of course, because the sheer amount of man-hours needed on all this are massive. Nor can you get away with cheap disposable labor for most of it, since writing specs and formal proofs and doing audits all requires a very skilled labor force.
Additionally, take supply chain steps like requiring independent approval for PRs and requiring the signing process to only work with automated builds. Don't allow for signing bits that were built on a dev machine.
Finally, I think it would help to start implementing signing executables and scripts which get checked at runtime. Windows obviously does executable signing, but it largely doesn't sign stuff like Python scripts. JS in the browser is kind of signed given that the whole site is signed via https. It's not perfect, but it would help in preventing one program from modifying another for very sensitive contexts like web servers.
You could probably still protect against this sort of attack using signing, but it would be much more laborious and annoying to get working. The idea is that you would somehow declare that OpenSSH binaries must be signed by a *particular* key/authority, that VS Code is signed by Microsoft, Chrome signed by Google, etc. Additionally, the config declaring all of this obviously needs to be secured so you'd need to lock down access to those files (changing key associations would need to require more permissions than just updating old software or installing new software to be useful).
The code in the tarball could have been prevented if only automated tarballs were permitted (for instance, GitHub's branch/tag source tarballs) or caught after the fact by verifying the file hashes in the tarball against those in the repo.
I thought it was in both, however the backdoor wasn't inserted with the default build time options, so it ended up in the tarball, but not if you just did make unless you set things up the same as the build server.
JiaT75 created the release on GitHub which involves creating a description and uploading tarballs. Their PGP key has been used to sign them since May 2023.
1. The repo contained two test xz files which contained the obfuscated exploit .o file
2. JiaT75 created the release on GitHub which allows you to upload your own tarball for the release.
3. The release tarball contained a modified build-to-host.m4 that extracted the exploit .o so it would be linked into the library if building on a Linux x86-64 machine.
4. Fedora/Debian/whoever pulled the release source tarball from GitHub and ran their builds for x86-64 Linux which caused the exploit .o file to be linked into the binary they then packaged up for release
It’s almost never recommended, in favor of curve 25519.
Malware authors frequently change the crypto magic numbers to prevent detection.
I'm especially interested in how such exploits can be prevented in the future.
>Lasse regularly has internet breaks and is on one at the moment, started before this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.
xz and the Debian distribtion of OpenSSH were targeted.
AI has been trying to solve that for several decades and in my experience in production these systems usually end up being a set of expert written rules saying "if X is greater than Y, notify someone to check".
Statistical AI systems are just naive induction, and so largely useless.
>Interestingly enough, "Jia Tan" is very close to 加蛋 in Mandarin, meaning "to add an egg". Unlikely to be a real name or a coincidence.
Tan is a real Chinese surname, and could be mandarin or hokkien.
Either way the name is likely made up. Obviously the attacker wasn't going to just use their real names.
If this backdoor can be classified as a crime, github logs can identify the IP/location/other details of the attacker which is more than enough to identify them, unless their OPSEC is perfect, which it almost never is (e.g. Ross Ulbricht).
1) Probably by design protonmail doesn't keep these kinds of logs around for very long
2) Hacking groups pretty much always proxy their connection through multiple layers of machines they've rooted, making it very difficult or impossible to actually trace back to the original IP
True regarding the second point.
[1]: https://techcrunch.com/2021/09/06/protonmail-logged-ip-addre...
Isn't this wild? Shouldn't the ciphertext be encrypted with an ephemeral symmetric key signed by the privkey? I guess anyone with the public key can still read any payload, so what's the point?
By tying it to a particular key owned by the attacker, no other party can trigger the exploit.
The only real improvement that I can see being made would be adding perfect forward secrecy so that a logged session couldn't be decrypted after the fact. But that would likely add a lot of complexity (I think you need bidirectional communication?)
> The only real improvement that I can see being made would be adding perfect forward secrecy so that a logged session couldn't be decrypted after the fact. But that would likely add a lot of complexity (I think you need bidirectional communication?)
Yeah.
20 years ago, I was working at an ISP/Telco and one of our vendors had a permanent admin account hardcoded on their gear, you couldn't change the password and it didn't log access, or show up as an active user session.
Always limit traffic to just what is necessary, does the entire internet really need to be able to SSH to your box?
This can turn into a footgun: you're away from your usual device, something happens and you desperately need to connect, but now you can't because all the devices in your vicinity don't have the ability to perform $SECURITY_FEATURE_X so that you can connect, and you're screaming at yourself for adding so much security at the expense of convenience.
This could happen as easily as restricting logins to SSH keys only, and not being able to use your SSH key on whatever host you have available at the time, wishing you'd have enabled password authentication.
And port knocking is one of the few effective methods against the standard distributed brute forcing slow attacks.
Note if you blanket-ban IN/RU/UK/CN/HK/TW/IR/MX/BZ/BA + tor/proxies lists, than 99.998% of your nuisance traffic issues disappear overnight. =)
These are very well documented tricks, and when combined with a standard port 22 and interleaved knock ports tripwire 5 day ban rules... are quite effective against scanners too.
I am currently on the clock, so can't write up a detailed tutorial right now.
Best regards, =)
The list of countries there: India, Russia, United Kingdom, China, Hong Kong, Taiwan, Iran, Mexico, Belize, Bosnia and Herzegovina
I’m amused that the UK is in that group.
It is apparently a free service they offer people even when told to get stuffed.
=)
My spastic ban-hammer active-response list i.e. the entire block gets temporarily black-holed with a single violation: AU BG BR BZ CN ES EE FR GB HR IL IN ID IR IQ JP KG KR KP KW LV MM MX NI NL PA PE PK PL RO RU RS SE SG TW TH TR YE VN UA ZA ZZ
If that's important, one should be able to set up port knocking such that you're able to do the knocks even by changing the port in a sequence by hand on e.g. a web browser address bar.
(Security is not my thing; don't judge me!)
~ % gpt4 'write a very short bash script that takes the number stored in ~/.ssh/knock_seq, increments it by 1 and saves it to the file. It then takes the new number and concatenates it with the value stored in the file ~/.ssh/secret. It pipes the resulting string to sha1sum, spitting out binary. It then takes both the resulting sha1sum and the number used and pipes their concatenation to "nc -u $host $(cat ~/.ssh/knocking_port)". be brief'
knock_seq=$(cat ~/.ssh/knock_seq)
let knock_seq++
echo $knock_seq > ~/.ssh/knock_seq
concat_seq_secret=$(echo -n "${knock_seq}$(cat ~/.ssh/secret)")
sha1_output=$(echo -n $concat_seq_secret | sha1sum -b | awk '{print $1}')
final_output="${sha1_output}${knock_seq}"
host=localhost
knocking_port=$(cat ~/.ssh/knocking_port)
echo -n $final_output | nc -u $host $knocking_portThis should prevent against replays. Throw in some rate limits somewhere maybe to not get DDoSed, especially if you let socat `fork`.
This problem solved by XRay in all of it's versions. It could be possible (if overkill) to use mostly same methods to authenticate correct user and provide eir access.
It means that people who can't intercept traffic can't talk to the ssh server-- and that's most attackers at the beginning phases of an attack. And even someone who can intercept traffic needs to wait for actual administrative activity.
Also underrated I think is security by simplicity. OpenSSH should be extremely simple and easy to understand, such that every proposal and change could be easily scrutinized. Cryptographic constructions themselves are almost mathematically proven invulnerable, then a small codebase can go most of the way to mathematically provable security (bonus points for formal verification).
But for this kind of system there's usually some kind of human vulnerability (e.g. system updates for your distro) in the loop such that the community needs to remain watchful. (It's fun to consider an application that's proven correct and doesn't need updating every again, but usually that's not practical)
Port knocking infrastructure can be minimal, knowing nothing but addresses knocking. It can also be completely outside the protected service on a gateway.
Indeed, it can even be no-code, e.g. https://www.digitalocean.com/community/tutorials/how-to-conf...
> OpenSSH should be extremely simple and easy to understand, such that every proposal and change could be easily scrutinized.
But OpenSSH intrinsically is going to have a much larger attack surface.
> then a small codebase can go most of the way to mathematically provable security (bonus points for formal verification).
It's worth noting this would not have helped against this attack:
* It was against another dependency, not openssh
* The actual vulnerability didn't occur in the code you'd inspect as part of verification processes today. (I don't think anyone is formally verifying build process).
Also, whenever some layers may allow compromising everything, the supply chains of the layers should be minimal or correlated (same supplier), to avoid increasing such supply chain risks.
And generally with depending on a device (phone, xxxkey or whatever) for access.
You can write down the full TOTP secret, and make new 'TOTP keys' whenever you want. Suggested apps: Firefox Extension named "TOTP", and Android App named "Secur". You don't need the heavier apps that want you to create accounts with their company.
Using a TOTP key only provides a little more protection than any other complex password, since you're not typing in the whole thing during a login attempt.
Then you attach a device that can have port knocking to that unsupported host. Also, I remember it was called port punching not knocking.
You can type http://hostname:porttoknock in a browser.
As long as you're not behind a super restrictive gateway that doesn't let you connect to arbitrary ports, you're golden.
I suppose I should have said it was a new-to-me use case for that feature.
First, I don't see how this works where there's a single server (e.g. colocation).
Second, doesn't that just make Wireguard the new hack target? How does this actually mitigate the risk?
The main benefit of port knocking is that it allows you to present your ports as normally closed. If you have a system and you are worried about it getting port scanned for whatever reason, it makes sense to have a scheme where the ports are closed unless it receives a knock. So if someone gets access they shouldn't for whatever reason and pulls a portscan off, the information they get about your system is somewhat limited.
In this scheme, it would be much better to have an access point you authenticate with and then that handles the port knocking for the other devices. So it is a kind of obscure method that is really only useful in a specific use case.
As far as SSH keys go, I would argue that SSH support is so ubiquitous, and SSH access is so powerful that it is a reasonable security tradeoff. I also don't think that SSH ports should be exposed to the internet, unless it is a specific use case where that is the whole point, like GitHub. I'm very down on connecting directly through SSH without network access. The xz situation has validated this opinion.
I personally don't know any application where you really need supreme admin access to every device, from any device, from anywhere in the world, while at the same time it has extreme security requirements. That's a pretty big task. At that point, constructing a dedicated, hardened access point that faces the internet and grants access to other devices is probably the way to go.
Did it? At the point an attacker has remote code execution, couldn't they just as easily pivot into an outgoing connection to some command and control server? I don't see how some intermediary access point would have alleviated this problem. If the call is coming from inside the house, the gig is already up.
The attacker doesn't have remote code execution in the xz case unless they can speak to your port 22. Port knocking prevents them from doing so, provided they don't know how to knock.
To me, that is what makes this hack so noteworthy. Hacker got access, and then kept playing the long game. Very spooky. And yes, someone capable of this probably has a plan for gaining network access if they had a target in mind. Even with best security practice, this rabbit hole goes pretty deep.
There's a lot of solutions now where the host has an agent that reaches out instead of allowing incoming connections which can be useful (assuming you trust that proxy service/software).
One place I worked, we ran our jumphost on GCP with Identity Aware Proxy and on AWS with SSM sessions so had to authenticate to the cloud provider API and the hosts weren't directly listening for connections from the internet. Similar setup to ZeroTier/TailScale+SSH
The "obscurity" part mostly serves to limit the noise of drive by and bot'd attacks, such that each attack attempt that you end up stopping and catching is a more serious signal, and more likely to be directed. It's about short circuiting much of the "chaff" in the signal such that you are less warning fatigued and more likely to seriously respond to incidents.
The obscurity is not meant to prevent targeted attacks.
In the past, sensitive networks relied heavily on obscurity alone. Just dialing the right phone number could grant access to highly sensitive networks. However, there's a cautionary tale of a security researcher who took the phrase 'don't do security through obscurity' to heart and dared hackers, leading to disastrous consequences.
Obscurity, when used appropriately, complements broader security measures. Take SSH, for example. Most bots target its default port. Simply changing that port removes the easy targets, forcing attackers to use more sophisticated methods, which in turn, leaves your logs with more concerning activity.
Indeed: I whitelist hosts/IP blocks allowed to SSH in. I don't use port-knocking but I never ever criticized those using port knocking.
I do really wonder if people are still going to say that port knocking is pointless and security theatre: we now have a clear example where people who were using port-knocking were protected while those who didn't were potentially wide open to the biggest backdoor discovered to date (even if it's wasn't yet in all the major distros).
> ... does the entire internet really need to be able to SSH to your box?
Nope and I never ever understood the argument saying: "Port-knocking is security theatre, it doesn't bring any added security".
To me port-knocking didn't lower the security of a system.
It seems that we now have a clear proof that it's actually helping versus certain type of attacks (including source-code supply chain attacks).
Who was saying that?
Yeah, it's a "straw man" pretending the person they are addressing was just planning on running telnet with port knocking.
So would have a vpn or using a bastion host with a custom non standard ssh implementation...
At some point you have to make the choice to not implement a security measure and I would argue that should stop at vpn+standard software for secure access.
If you are a bigger company, probably add SSO and network segmentation with bastion hosts and good logging.
Port Knocking doesn't add any security benefit in the sense that there are known non avoidable security risk aka your transmit your password(knocking) in clear text over the network.
You also add another program with potential vulnerabilities, and as port knocking is not as popular as e.g. sshd, wireguard, maybe it gets less scrutiny and it leads to a supply chain attack?
Security measures are also not free in the sense that somebody has to distribute them and keep the configuration up to date, even if that person is you, that means syncing that connect-to-server-script and keeping it in a secure location on your devices.
You also add another program with potential vulnerabilities, and as port knocking is not as popular as e.g. sshd, wireguard, maybe it gets less scrutiny and it leads to a supply chain attack?
That other program is just a stateful firewall, aka the Linux Kernel itself. If you can't trust your kernel then nothing you do matters.Granted, that program is really small and could be easily audited, but that same time could have been spent on trying apparmor/seclinux + a good vpn and 2fa
To some extent knockknock also trusts the Python interpreter which is not ideal (but maybe OK)
https://wiki.nftables.org/wiki-nftables/index.php/Port_knock...
This take is bordering on not even wrong territory. The point of port knocking isn't to increase entropy of your password or authentication keys per se, it is to control who can send packets to your SSH daemon, either to limit the noise in your logs or to mitigate an RCE in the SSH daemon. The vast majority of potential attackers in the real world are off-path and aren't going to be in a position to observe someone's port-knocking sequence.
Is VPN a better solution? Maybe, but VPNs, especially commercial VPNs have their own set of challenges with regard to auditability and attack surface.
If you read my full sentence in the context it stands, I argue that authorizing access to your openssh instance is done by sending an authentication code in cleartext.
It does not matter if that authentication code is in the form of bits, tcp pakets, colors or horoscopes as long as your transmit that in clear text it is in fact no a secure mechanism.
Yeah but now you basically have to always run an vpn that only exists between your server and your client, because otherwise your clear text authentication code is visible and at that point just use wireguard and make a 1-1 tunnel only for ssh with no known attacks, even if the attacker is in the same network.
Yes a vpn that has no known reliably attack vector is definitely better than a protocol with a known working attack vector
I did read your full post. Your claim that port knocking adds no security benefit is just simply incorrect. Even if you take for granted the scenario of an attacker who can recover the port knock sequence, there is still the benefit of shielding the SSH daemon from other attackers who cannot do so. Which is…a “security benefit.”
Describing the port knocking sequence as an extension of someone’s authentication key is where this goes off into not even wrong territory. A dynamically controlled firewall rule (port knocking) is not fungible with bits from an authentication token for the reasons I’ve already outlined - the benefit is limiting network access to the daemon at all.
> Yes a vpn that has no known reliably attack vector is definitely better than a protocol with a known working attack vector
You talk about recovering someone’s port knocking sequence as if that is trivial to do or in any way reliable. It is, in fact, neither of those things. An attacker would have to either:
1) sniff network traffic in front of the server
2) sniff network traffic in front of the client
3) compromise the server
4) compromise the client
5) brute force the port knocking sequence without getting locked out.
Most attackers are going to be in a position to try brute forcing — that’s it.
Meanwhile, you may not have noticed, but commercial VPNs have suffered a steady stream of high-impact CVEs for the last few years.
Wireguard is certainly better in this regard than any commercial VPN I know of, but it does have challenges with regard to key distribution/device enrollment, and thus from a practical standpoint limits access to pre-enrolled endpoints, which is a limitation someone using port knocking does not have.
-networking.firewall.allowedTCPPorts = [ 22 ];
+networking.firewall.interfaces."tailscale0".allowedTCPPorts = [ 22 ];
I probably should have done this ages ago.Worse, most corporate, public wifi, etc networks block all sorts of ports. So at home sure you can open random ports but near everywhere else its just 80 and 443. Now you can't knock. But your https vpn works fine.
Also a lot of scary stuff here about identity and code checkins. If someone is some contributor, how do we know if their creds havent been stolen or they've been forced via blackmail or whatever to do this? Or how many contributors are actually intelligence agents? Then who is validating their code? This persons code went through just fine, and this was only caught because someone noticed a lag in logins, which by then, is a running binary.
FOSS works on the concept of x amount of developer trust, both in code and identity. You can't verify everyone all the time (creds and certs get stolen, blackmail, etc), nor can you audit every line of code all the time. Especially if the exploit is submitted piecemeal over the years or months. That trust is now being exploited it seems. Scary times. I wonder if how FOSS works will change after this. I assume the radio silence for Theo and Linus and others means there's a lot of brainstroming to get to the root of this problem. Addressing the symptom of this one attack probably won't be enough. I imagine some very powerful people want some clarity and fixes here and this is probably going to be a big deal.
I wouldn't be surprised if a big identity trust initiative comes out of this and some AI stuff to go over an entire submitter's history to spot any potentially malicious pattern like this that's hard for human beings to detect.
You can if you distribute this job among volunteers or hire people to do that. There are millions of developers around the world capable to do this. But reality is that nobody wants to contribute time or pay for free software.
/s
The security of port knocking is also woefully inadequate:
• It adds very little security. How many bits are in a “secret knock”?
• The security it does add is bad: It’s sent in cleartext, and easily brute-forced.
• It complicates access, since it’s non-standard.
1. <https://en.wikipedia.org/w/index.php?title=Kerckhoffs%27s_pr...>
Strict firewalls, VPNs, and defense-in-depth is really the only answer here.
Of course, those things all go out the window too if your TCP stack itself is also compromised. Better to just air-gap.
However, in this case, the two-layer approach would not be a protection. Sure, in this case the SSH daemon was compromised, and a VPN before SSH would have protected SSH. But what if the VPN server itself was compromised? Remember that the SSH server was altered, not to allow normal logins, but to call system() directly. What if a VPN server had been similarly altered? This would not have protected SSH, since a direct system() call by the VPN daemon would have ignored SSH completely.
It is a mistake to look at this case and assume that since SSH was compromised this time, SSH must always be protected by another layer. That other layer might be the next thing to be compromised.
I don't know how VPNs are implemented on Linux, but in principle it should be possible to sandbox a VPN server to the point where it can only make connections but nothing else. If capabilities are not enough, ebpf should be able to contain it. I suspect it will have full control over networking but that's still very different from arbitrary code execution.
So if someone cracks that box, their access is limited, and they still need to make a lateral move to access actual data.
In the case of this backdoor, if you have SSH exposed to the internet on a locked down jump box, AND use it as your internal mechanism for accessing your valuable boxes, you are owned, since the attacker can access your jump box and then immediately use the same vulnerability to move to an internal box.
In the case of a hypothetical VPN daemon vulnerability, they can use that to crack your jump box, but then still need another vulnerability to move beyond that. Not great, but a lot better than being fully owned.
You could certainly also accomplish a similar topology with two different SSH implementations.
Kerckhoff's law goes (from your link):
> The principle holds that a cryptosystem should be secure, even if everything about the system, except the key, is public knowledge.
With this backdoor, that principle does not hold; it's utterly irrelevant. Obscuring the fact that there even is a ssh server to exploit increases safety.
(I dislike port knocking because of your third point, that it complicates access. But I don't think your assertions about its security principles hold water, at least not in this case.)
(Ultimately, though, instead of something like port knocking or even running sshd on a non-standard port, if I wanted to protect against attacks like these, I would just keep sshd on a private network only accessible via a VPN.)
If we’re talkning about this specific backdoor, consider this: If the attacker had successfully identified a target SSH server they could reasonably assume had the backdoor, would they be completely halted by a port knocker? No, they would brute-force it easily.
Port knocking is very bad security.
(It’s “Kerckhoffs’s <law/principle/etc.>”, by the way.)
> If you want more secret bits which users need to know in order to access your system, increase your password lengths, or cryptographic key sizes.
If your sshd (or any other exposed service) is backdoored, then the "effective bits" of any cryptographic key size is reduced to nil. You personally cannot know whether or not your exposed service is backdoored.
Bottom-line is: adding a defense-in-depth like port knocking is unlikely to cause harm unless you use it as justification for not following best-practices in the rest of your security posture.
That way it would not be in clear text, and the number of bits of security will be order of magnitude larger even with very simple password. The public facing sshd can also run more lightweight algorithms and disable loggings for lower resource usage.
Regardless if one uses two sshd or port knocking software, the public facing daemon can have backdoors and security bugs. If we want to avoid Xz-like problems then this first layer need to be significant hardened (With SELinux as one solution). Their only capability should be to open the second layer.
Building security in depth correctly is not simple. It takes work to construct layers so that one compromised layer do not cause whole system failure.
Also as pointed out elsewhere, modern port knocking uses Single Packet Authorization which allows for more bits. It's also simpler and uses a different mechanism than ssh (which due to its age, has historically supported a bunch of different login and cryptography techniques), which reduces the chance that an attacker would be able to break both.
"Play stupid games, win stupid prizes"