We updated our RSA SSH host key
github.blog
github.blog
Github's customers trust GH/MS with their code, which, for most businesses, is a high value asset. It wouldn't surprise me if this all results in a massive lawsuit. Not just against GH as a company, but also to those involved (like the CISO). Also, how on earth was it never discovered during an audit that the SSH private key was plain text and accessible? How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM?
Obviously, as a paying customer I am quite angry with GH right now. So I might be overreacting when I write this, but IMO the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license.
[0] https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?
Not just nation state actors, but basically anyone in a position to MITM.
Also, you don't have to be a nation state actor to extort a GH employee. Any bad guy can do a "give me this key or I'll hurt your kid". People are being extorted for a lot less.
There are billions of dollars of assets flowing through GH's infrastructure, for the sake of safety (!= security) of Github's employees, nobody should ever have access to key material.
Do you mean source code here? I have a hard time believing source code holds that much value.
If there are ~350 million private repos then they'd only need to be worth an average of $30 each to be worth a billion dollars in total. Which doesn't seem farfetched.
For proof, try searching for a mundane string in GH Code search. The vast majority of repos you see will be basically garbage.
The host key is the only thing ensuring you’re actually talking to GitHub.com when you push code.
To add to sibling comments, it should not have been possible to make this mistake. That it was possible is concerning.
Great! Then I can communicate confidentially with whomever is MITM'ing me.
/s
In fact, this is not a supported option in openssh.
You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions.
> In fact, this is not a supported option in openssh.
This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.
Agreed. I have seen some crazy stuff in the payment card industry. I can't recall what I can and can't talk about so I'll just say "Atalla".
Edit: Apparently OpenSSH's sshd also supports the SSH agent protocol for host keys, and ssh-agent does support PKCS#11 – so I stand corrected!
How can openssh documentation be vendor-specific?
Or are you saying that vendors commonly provide an openssh fork/patchset/plugin allowing for HSM-resident host keys?
I isn't, because the cryptography is (in case of HSM) not handled by OpenSSH itself. So OpenSSH's configuration has nothing to do with the HSM.
Usually, the actual cryptographic functions are not performed user-space, but handled by the kernel, which in turn can offload this to dedicated hardware. Basically if you compile OpenSSH for it to use kernel level cryptographic function, then OpenSSH can work with a HSM without it even knowing it.
Disclaimer: this is simplified explanation, there is a lot more to this, and I am by no means an expert on this matter.
Edit: meant to say kernel level cryptographic functions, not TLS.
I‘m aware of the PKCS#11 integration in the OpenSSH client and have dabbled a bit with it but was not aware of any server side equivalent.
And how does TLS fit in here? SSH is a very different protocol from that, no?
Update: I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support or kernel-mode cryptography (which also in the case of Linux only addresses symmetric encryption to my knowledge, at least the mainline kernel version I know of).
Maybe you're thinking of some other SSH implementation? The protocol definitely allows for server-side HSM usage, and Github at their scale is not bound to OpenSSH in any way.
It makes a lot of sense, since it avoids having to link the HSM/PCKS#11 stuff against sshd.
OpenSSHd talks to an ssh-agent that then talks to the HSM:
> Identifies the UNIX-domain socket used to communicate with an agent that has access to the private host keys. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be read from the SSH_AUTH_SOCK environment variable.
There is the HostKeyAgent configuration directive, which communicates over a unix domain socket to make signing requests.
https://framkant.org/2017/10/strong-authentication-openssh-h...
https://github.com/openssh/openssh-portable/blob/12492c0abf1...
AIUI OpenSSH does not provide any way to use PKCS11 to protect host keys, which are the concern here.
You can use PKCS11 to sign OpenSSH certificates, so if GitHub had elected to use certificates here, it could have protected the CA keys for those certificates in an HSM, but it did not.
https://framkant.org/2017/10/strong-authentication-openssh-h...
It looks like they currently use the `ssh_bind_options_set` function with `SSH_BIND_OPTIONS_HOSTKEY` to set the host keys which means they exist on disk at some point. HSM aside, I believe it would be possible to use the `ssh_bind_set_key` and deserialize them from a secret vault so they only exist in the memory of the ssh daemon.
Obviously they also just straight up have enough resources to fork the code and modify it to use an HSM.
Source: looking at their ssh server portion of `babeld` in ghidra right now as part of hunting for bug bounties.
Do you realize that the code just sits on GitHub servers even if it's private?
If you have any degree of paranoia, why do you put your code into GitHub?!?!
Like, if you work on code which
ISO/IEC 27001:2013 doesn’t say you have to store private keys in HSMs? It just requires you to have a standards compliant ISMS that implements all Annex A controls and all of the clauses. Annex A and the clauses don’t specifically mandate this.
If you can convince an auditor that you have controls in-place that meet the standard for protecting cryptographic media you basically meet the standard. The controls can be a wide variety of options and don’t specifically mandate technical implementation details for many, many things.
You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Just because your auditor would red flag you doesn’t mean all auditors would. The standard is only as effective as the weakest, cheapest auditor, and there are perverse incentives that make auditors financially incentivized to certify companies due to recurring revenue.
Thanks for the insight, good advice.
But also from the same GH article [0]:
The ISO 27001 certification is the latest addition to GitHub’s compliance portfolio, preceded by SOC and ISAE reports, FedRAMP Tailored LiSaaS ATO, and the Cloud Security Alliance CAIQ.
Do you have any knowledge on one of these certifications (for exmaple FedRAMP) that puts any restrictions on handling key material?
[0] https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
It took us ~2 years of back & forth with various parties & auditors to get per-client exclusions for accepting end-customer debit PIN codes in-branch on an iPad screen. These banks do not have fully-compliant solutions and must have exceptions on file.
As the adjacent commentor 1970-01-01 states, PCI DSS is actually pretty strict and requires use of HSMs. However, that level of PCI compliance is only required for institutions that actually handle the full credit card number. GitHub uses Stripe as a payment gateway, so they don't need to meet it.
FIPS 140-3 levels 3 and 4 often are met by using HSMs, but technically speaking there aren't any standards that I know of that exist outside of the payments industry that have hard requirements for HSM use for all cryptographic key media.
I think the unfortunate reality is that many organizations struggle to deploy HSMs widely for all cryptographic media because they aren't very scalable and deployment can lead to other operational constraints that many companies can't or won't deal with. It's much easier for low-frequency high-importance tasks like signing new releases of OS images or packages, rather than I/O heavy high-frequency operations for a site like GitHub.
So, in-summary, look for PCI DSS or FIPS 140-3 Levels 3 and 4, but, be prepared to discover that "creative" solutions may let a company meet even the highest levels of FIPS compliance without HSMs for all cases.
I know it's sucky advice, but for ISO in-particular, I suggest just acquiring and reading 27001 if you want to use it as a basis for decision making. It does offer a lot, and I think it's a very well-written standard, but all of the implementation advice is in ISO 27002, and it's not required. The advice in 27002, when used to meet 27001, leads to a very compelling program. But a 27001-compliant org can completely forego 27002 and DIY it as long as it meets the test criteria set-out by the auditor.
Edit: Also, every auditor will include standard disclaimers in their report that they perform sampling-based testing, and that the testing is not a complete guarantee of the state of the company. ISO in-particular is performed by getting an initial audit, followed by two years of surveillance audits that test the entire suite of controls. But due to sampling-based methodology, something can be overlooked or evidence can be provided that doesn't holistically reflect the org in all cases. If it's any consolation, this particular issue will certainly be in their audit for next year, as a security incident and probably lead to an opportunity-for-improvement from the auditor.
So you may be able to get certified for policy, process but not necessarily the programming to allow something in the first place.
Hopefully now knowing this will improve under Microsoft.
For those who value this, self- hosting or on-prem instances of git might shoot up in importance.
ISO 27001 certification does not require you to put keys into an HSM. The standard requires you to have controls in place, be aware of your risks and to maintain a risk register. But in no way does the standard require HSM's.
The standard would even be OK with storing this on a floppy drive if the risks surrounding that were identified and mitigated (or accepted).
Not to diminish the problems of having a large entity like Github handle a private key like that, but if that was your level of paranoia, you probably should have used commit signatures all along and not relied on Github to do that job for you.
I also don't want to diminish the concerns around Github or similar orgs losing control of a private key, but the far more realistic concern for the vast majority of threat models is often put to the wayside in favor of what amounts to a scary story. Rather than the straightforward key removal and replacement that this should be, I (and surely many others) have spent all morning combatting this specific FUD that cropped up on HN with leadership and many engineers. It's actually quite detrimental to quickly remediating the actual concerns introduced by this leak.
I understand that security inspires people to be as pedantic as possible - that's where some big exploits come from on occasion - but I really hope the average HN narrative changes toward "what is your actual, real-world threat model" vs. "here is a highly theoretical edge-case scenario, applicable to very few, that I'll state as a general fact so everyone will now wonder if they should spend months auditing their codebase and secrets". Put simply: this is why people just start ignoring security measures in the real world. Surely someone has already coined the term "security fatigue".
It's all just a bit unbalanced, and definitely becomes frustrating when those suggesting these "world is burning" scenarios didn't even take the available precautions that apparently would satisfy their threat model (i.e. commit sigs, as you suggested)
Ok, end rant :)
Having a correct threat model is the first step towards building reasonable security controls. But far too many are willing to pander to the "It rather involved being on the other side of this airtight hatchway" [0] scenarios.
[0] https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
1) Nation state level actors can probably insert or compromise high level staff, or multiple high level staff, at any given company, and perform MITM attacks fairly easily. And some could compel turning over your code or secrets more directly anyway. Not worth worrying about this scenario: nobody working on anything truly sensitive should be using any externally hosted (or even externally networked) repositories.
2) It is much more difficult for other actors to do a MITM attack, and if they did, they'd probably have access to your code more directly.
3) Your code actually isn't worth much to anybody else. Imagine someone launching a complete clone of HN or any other site. Who cares? Nobody. What makes your code valuable is that you have it, and that you have a network and relationship with your customers. If somebody stole my company's codebases, I'd feel sorry for them, that they are going to waste any time wading through these useless piles of shit. The only potential problems are if secrets or major vulnerabilities are exposed and provide a path for exploit (like ability to access servers, services, exposing potential ransomware attacks).
This leak opened a time window big enough for that to happen. We may or may not know if it did. I doubt this info would be offered to the public because it would sink the business.
The cloud is always the convenience of someone else’s computer over some amount of security.
Requiring all commits to be signed by trusted keys avoids the risks associated with someone tampering with a repository hosted on GitHub if they are able to get access to it, although it doesn't protect against code being leaked.
See here for details: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
The risk of disclosure is pretty obvious with GitHub, and I’d assume anyone with low risk tolerance here is using something else, including the on-prem GitHub. I can think of a dozen higher risks.
Many signed commits ("Verified" in green) on GitHub are signed with GitHub's own private key¹, not the committer's key. There's a good technical reason, but many committers don't realise their own signing key isn't the one used on their signed commits to the main branch.
That GitHub private key would be a fun one to have leaked!
It would invalidate most of the "Verified" flags on commits on most repo main branch histories.
(¹ GitHub uses GitHub's own commit-signing key when you use the GitHub GUI to merge or squash-merge commits if you already signed them, and the resulting commits show as "Verified" the same as conmits signed by your own key. So you do have to sign with your own key,, but it's not the key used on the main branch commits everyone sees whrn they visit and doenload. Many controlled workflows require approved PRs to be merged to main/master through GitHub's tools, and many users default to using the GUI for PR merges by choice.)
Number of people just not blindly using TOFU with github over ssh must be quite low.
Who here went here https://docs.github.com/en/authentication/keeping-your-accou... and added the keys manually to known_hosts before using github for the first time?
Oh, I missed the "not" the first time I read it, which changed your overall meaning entirely. My bad.
I think this suggests we need more information from github. For instance GH employees may not always have had live access to this key, this could have happened as part of an operation that gave temporary access to an employee only recently. Or it could have been stored plaintext on multiple employees' home computers since creation.
When was the leaked key created anyway?
I suppose (Open?)SSH's PKI mode could support a model like that, but as others have noted here, this requires much more manual work on the user's side than comparing a TOFU key hash.
Maybe that model could be extended to allow TOFU for CAs, though? But I think PKI/CA mode is an OpenSSH extension to the SSH protocol as it is, and that would be a further extension to that extension...
Storing the key in some kind of credential vault that can only be accessed from the hosts that need it at startup would usually be enough to prevent this particular kind of error (unless you're giving root on those boxes to people without enough sense to avoid checking private keys into git, in which case you've probably got worse problems).
Regarding the secrecy of the data you host at github, you should operate under the assumption that a sizeable number of github employees will have access to it. You should assume that it's all sitting unencrypted on several different disk platters replicated at several different geographically separated locations. Because it is.
One of the things that you give up when you host your private data on the cloud is controlling who and under what circumstances people can view or modify your private data. If the content of the data is enough to sink you/your company without remedy you should not store it on the cloud.
[1] Which is sort of tautological and silly, because that's true of all sites and all host keys. What the comment was trying to imply was that the choice of storage of this key invalidates any trust we might have in GitHub/Microsoft regarding key management, and that therefore we shouldn't trust them. Which is also tautological and silly. Either you trust them or you don't, that's not a technological argument.
When we signed on with GH as a paying customer over a decade ago, they were quite clear that private repos should not be considered secure storage for secrets. It’s not encrypted at rest, and GitHub staff have access to it. It takes only a few clicks to go from private to public.
The real question is how many services are able to access the data live and how many support and debug interfaces (indirectly) allow you to read it. Measure GitHub's success in securing the secrecy of private repos in how few employees can breach it without causing alarms. Even without cynicism I would be surprised if it was their main concern. Data integrity is far more important for code. (There are notable exceptions, of course. If applicable, don't put it in the cloud!)
There are plenty of tools to either keep them encrypted (we just use simple GPG, but our team is small) or just auto-generate and never show to user in the first place (various key vaults that can be used from automation like HashiCorp's Vault)
If ssh token based auth is used, it's different of course, because then the server gets access to the token. Ideally Github would invalidate all their auth tokens as well.
The fun fact is that a token compromise (or any other attack) can still happen any point in the future with devices that still have outdated ssh keys. That's a bit unfortunate as no revocation mechanism exists for ssh keys... ideally clients would blacklist the ssh key, given the huge importance of github.
E.g. keybase could do X.509 like Certificate Transparency.
:
$ keybase pgp -h
NAME:
keybase pgp - Manage keybase PGP keys
USAGE:
keybase pgp <command> [arguments...]
COMMANDS:
gen Generate a new PGP key and write to local secret keychain
pull Download the latest PGP keys for people you follow.
update Update your public PGP keys on keybase with those exported from the local GPG keyring
select Select a key from GnuPG as your own and register the public half with Keybase
sign PGP sign a document.
encrypt PGP encrypt messages or files for keybase users
decrypt PGP decrypt messages or files for keybase users
verify PGP verify message or file signatures for keybase users
export Export a PGP key from keybase
import Import a PGP key into keybase
drop Drop Keybase's use of a PGP key
list List the active PGP keys in your account.
purge Purge all PGP keys from Keybase keyring
push-private Export PGP keys from GnuPG keychain, and write them to KBFS.
pull-private Export PGP from KBFS and write them to the GnuPG keychain
help, h Shows a list of commands or help for one command
$ keybase pgp drop -h
NAME:
keybase pgp drop - Drop Keybase's use of a PGP key
USAGE:
keybase pgp drop <key-id>
DESCRIPTION:
"keybase pgp drop" signs a statement saying the given PGP
key should no longer be associated with this account. It will **not** sign a PGP-style
revocation cert for this key; you'll have to do that on your own.
- "Revoked a PGP key, is there any way to get a revocation certificate now?" https://github.com/keybase/keybase-issues/issues/2963
- "Overview of Certification Systems: X.509, CA, PGP and SKIP"
...
- k8s docker vault secrets [owasp, inurl:awesome] https://www.google.com/search?q=k8s+docker+vault+secrets+owa... https://github.com/gites/awesome-vault-tools
- Why secrets shouldn't be passed in $ENVIRONMENT variables; though e.g. the "12 Factor App" pattern advises to parametrize applications mostly with environment variables that show in /proc/pid/environ but not /proc/pid/cmdline
W3C DID supports GPG proofs and revocation IIRC:
"9.6 Key and Signature Expiration" https://www.w3.org/TR/did-core/#key-and-signature-expiration
"9.8 Verification Method Revocation" https://www.w3.org/TR/did-core/#verification-method-revocati...
Blockerts is built upon W3C DID and W3C Verified Credentials, W3C Linked Data Signatures, and Merkel trees (and JSON-LD). From the Blockerts FAQ https://www.blockcerts.org/guide/faq.html :
> How are certificates revoked?
> Even though certificates can be issued to a cohort of people, the issuer can still revoke from a single recipient. The Blockcerts standard supports a range of revocation techniques. Currently, the primary factor influencing the choice of revocation technique is the particular schema used.
> The Open Badges specification allows a HTTP URI revocation list. Each id field in the revokedAssertions array should match the assertion.id field in the certificate to revoke.
Re: CT and W3C VC Verifiable Credentials (and DNS record types for cert/pubkey hashes that must also be revoked; DoH/DoT + DNSSEC; EDNS): https://news.ycombinator.com/item?id=32753994 https://westurner.github.io/hnlog/#comment-32753994
"Verifiable Credential Data Integrity 1.0: Securing the Integrity of Verifiable Credential Data" (Working Draft March 2023) > Security Considerations https://www.w3.org/TR/vc-data-integrity/#security-considerat...
If a system does not have key revocation it cannot be sufficiently secured.
Well, at least not without encryption that is under your control.
By no means do I want to see the dev get fined or blackballed from the industry.
But if there is any 1 person responsible, it’s the person who did it. The reason why the dev shouldn’t be fined/blackballed is because it’s not just 1 person’s fault. I mean, fining or booting his manager out of the industry? Really?
"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."
Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
Why would that be unknown? GitHub has HTTP and SSH access logs, right?
Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key?
The answer is 31337.
To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
- if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable.
- rotating the key is the only correct reaction to that
- they should have better communications in place to help users mitigate MITM
- there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly
- on principle this speculation makes sense, but understanding the threat makes it moot
- you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
Thus rotating is the only logical course of action.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.
You just ignore it and hope for the best ?
Only if you are certain (and better be really sure you haven't missed any cache/cdn, temp files backus etc.) it wasn't accessed you do nothing.
You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.
I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background.
Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs.
Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
Blockchain is a perfect solution to this. I wonder why it is not considered yet.
Also it's way better than TOFU, you can just add the CA key to known_hosts and avoid TOFU for each machine.
(Nevermind that you'll probably not accidentally commit some semi-ephemeral host key that's rotated often somewhere, because it will not be some special snowflake key you care about, but something handled by your infrastructure software automatically for each machine)
How easy is it to rotate the keys of your CA?
I hope you don't need that SSH connection to fix your broken CRL endpoint!
We need to compare the relative risks within the same context, namely within a company like GitHub
So it's not relevant to bring up failures of other CAs
Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
We also have a way to put SSH host key fingerprints in DNS records already.
I'd guess that most systems aren't using DoH/DoT or end-to-end DNSSEC yet. Some browsers do, but that doesn't help tooling frequently used on the command line.
I suppose you could just accept X.509 certificates for some large/enterprise git domains, but that pokes up the hornet's nest that is CA auditing (the browser vendors are having a lot of fun with that, I'm happy that the OpenSSH devs don't have to, yet).
And where do you maintain the list that decides which hosts get to use TOFU and which ones are allowed to provide public keys? Another question very ill-fitted for the OpenSSH dev team.
There could be a well known URL that enables people to fetch ssh keys automatically in a safe manner.
So that any large entity can own your servers with easy. (Well, they already can, but not through this vulnerability.)
Anyway, the only thing CAs do is to move that prompt into another, earlier time. It's the same prompt, the same possibility for MITM, and the same amount of shared trust to get wrong. You just add a 3rd party that you have to trust.
SSH does have a CA system. Anybody that isn't managing a large datacenter will avoid it, for good reason.
Eh, let's not pretend existing SSL certificate validation is anything to write home about.
Even without any ephemeral servers involved, barely anybody is validating cert fingerprints on first use.
And among people using ephemeral servers, 99% of applications have either baked a certificate into their image (so that any compromised host means a compromise of the critical, impossible-to-revoke-or-rotate key) - or every new server gets a new cert and users have either been trained to ignore certificate warnings, or they've disabled strict host key checking and their known hosts file.
The existing SSL cert validation options are perfect if you're a home gamer or you're running a few dozen bare metal servers with all your SSL users within yelling distance in the same office. But we all know it's a joke beyond that.
There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
edit: I found this helpful and honestly had no idea I should be doing this (I’m a hobbiest not a professional) https://bitlaunch.io/blog/how-to-check-your-ssh-key-fingerpr...
The problem here is that their first command that they advise using is the one that removes the old key and most users are just going to stop right there because it solves the problem of getting the key warning.
The right solution here is to provide a command that most users are going to copy and paste that deletes the old key and adds the new key all at once.
If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM) because Github, their college or random person on internet said that that's what you have to do to get rid of error.
This is less likely because unlike for TOFU the SSH client just rejects the mismatch and insists you take manual action, and the likely manual action will be "Paste this stuff from the announcement".
So an adversary needs to either subvert whatever messaging you see (which is tricky, likely impossible for a random user visiting the github web site wondering what's wrong) or hope that you just try to muddle along and do TOFU again, putting you in the same spot as a whole bunch of users every day at GitHub.
If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame.
If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't they verify the key out of band?
The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
One way to solve this in TOFU is to have a time window where both keys are presented.
well, yes
github doesn't accept https push any more
Maybe you’re referring to how they no longer accept passwords for HTTPS auth? You have to auth for HTTPS push with a personal access token.
I get a bit paranoid when having to deal with Tokens on various CI/CD environments as it stands. And the things that start breaking every year when I forget to update them. Note: this is personal/hobby projects, not corporate stuff, where I'm strictly in the codebase and try to keep my fingers out of CI/CD beyond getting a Docker image built, and someone else configures the keys/auth.
I sure did. Doesn’t everyone?
So MITM for some of 50m users is strictly better than MITM for all of 50m users.
How do you know this?
Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
> leaked in public repo
Me: Yeah, that's why they are doing it.
What do I have to do to ensure I connected to the right server?? I thought just making sure the correct RSA key was in known_hosts would be enough?
If you read the blog post on a verified domain and saw the new key and updated manually, or you deleted the known key and verified the key fingerprint when it warned you about an unknown key, you should be good to go. Here, you trust the people who issue TLS certificates and you trust github to be in control of their domain name, so you can be reasonably confident that the key they advertised on their website is the correct key. If your internet connection was compromised, you would have got an error message when you connected to https://github.blog (because they wouldn't have a certificate from a trusted certificate issuer) or when you connected to the git server (because they wouldn't have the key you just trusted).
If you saw the blog post and then removed the old key and told ssh to save the new key it's receiving without checking it matches the value on the webpage, you might have a problem. The connection to github's ssl could have been compromised, and if you just accepted whatever it told you, you have no trusted intermediate to verify that the key is trustworthy. All you know is that each time you connect to github's server hereafter, you're either connecting to a server with the same key (no error), or you're connecting to one that doesn't have the same key (error message). But whether you can trust that key? You don't know that. You just know it's the same key.
But even if you did the latter, all is not lost. You can look at the known_hosts file (on Linux and MacOS it's ~/.ssh/known_hosts) and check the fingerprint. If it's what they advertise, then you're good to go. If it's different, you should fix it and find people who can help you deal with a security incident.
The reason people are raising a flag is that today, lots of people will be rotating their key. That means if you're looking to target someone, today is the day to do it. Even if 90% of people do it the proper way, by manually verifying the key, that still means there's going to be a lot of people who could be victimised today.
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s (RSA)
SHA256:br9IjFspm1vxR3iA35FWE+4VTyz1hYVLIE2t1/CeyWQ (DSA - deprecated)
SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM (ECDSA)
SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (Ed25519)We humans really aren't cut out for this, are we.
Generally speaking, one would not consider an internet comment directing folks to GitHub's actual SSH fingerprints a "man in the middle" as the phrase in this context usually has a negative implication, where in this case defanor is in fact simply mirroring the actual information that GitHub has officially posted in a way that is much more helpful than yetanotherjosh's "double check it is the expected value". For most of us idiots, we don't know what the expected value is!
So thank you defanor for sharing, and thank you darthrupert for asking for clarification. Y'all contributed to educating myself and others and now we know more because of it.
Good clarifications everywhere, yes.
(Not genuinely concerned about this risk, but "Reflections on Trusting Trust" reverberates.)
Not all of us are familiar enough with the SSH protocol to understand how to "double check the expected value"? Where can I determine what the expected value should be?
It should error like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
It would just take many ages of the universe, at present, to calculate a collision, right?
If you somehow can MITM an SSH connection on the first connection, you can probably use any key. Most people don't check the fingerprint.
But you are correct, computing an SSH key with a collisionwis expected to take an infeasible amount of time/energy with current understanding of crypto and available computers.
Or the parts below it about updating and verifying in other ways.
https://docs.github.com/en/authentication/keeping-your-accou...
Yes, this assumes the github-hosted docs and your SSL connection to them are not also compromised, but it's far better than not checking at all.
We should have a blockchain for certificates btw. That would be such an amazing solution to this problem. You could advertise ahead of the time that you are changing certificates and we could verify that it was in fact you.
I assume it's safe because the SSL cert for docs.github.com is probably not compromised, so it's giving us the right key, and compromising docs.github.com would be extra effort and is unlikely to happen.
However, I wonder what kind of steps an MITM attack would have to perform, I assume one of the easiest would be compromising my local DNS server, since regular DNS does not offer a high level of security, then github.com resolves to the attacker's IP and the attack works. Do you have examples of such attacks that don't involve a virus already active on the end user's PC? Maybe if someone owns an IP previously owned by Github that is still somehow advertised as being Github by some DNS lagging behing?
The best practice is to verify the fingerprint out of band using a secure channel. In this case, that's HTTPS and docs.github.com. If (hypothetically) docs.github.com was also compromised, then you don't have a secure channel.
https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack has some MITM examples.
It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere.
With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress.
This key held enough power over value that some might have used violence to acquire it.
With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public.
I have no words for this level of gross negligence.
Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan.
I will just leave this here: https://sfconservancy.org/blog/2022/jun/30/give-up-github-la...
This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection.
You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.
PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory.
You can totally take a single host key scheme and use HSMs or TEEs at scale.
Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM.
I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.
E.g. Foo can encrypt the key X with Bars public key and then Bar can import key X.
There's products out there that allow for keys to be securely shared amongst multiple HSMs, without the key ever existing in clear text outside the HSM.
Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today.
At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this.
There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.
Yep. Well, certificates exist exactly to bridge the GP's requirement with your reality.
In reality, the vast majority of users don't pay attention to SSH host keys at all.
Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target.
Next, they have to decide what they want to do.
If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult.
If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo.
And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have.
So I think your anger here is a bit over the top and unnecessary.
I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.
Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config.
If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical.
Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.
I mean, think of the confluence of things that have to line up for this to work for someone. Many stars have to align in order for someone to successfully exploit this leak. Is it impossible? No, of course not, and so GH was right to rotate the key here, even if their server request logs suggested that no one had accessed it.
If people actually have this sort of attack in their threat model, there are ways to protect against it. Signing commits and verifying them. Pinning to a particular git SHA1 hash. Etc. If people are not doing these things, then it's possible they've made the decision not to worry about this sort of attack. Sure, you can disagree with, but I think you'd probably be in the minority. That's ok, though; you can certainly protect the things you're responsible for in stronger ways.
A wildcard TLS cert or an SSH host key in plaintext is a loaded weapon laying around and it will be used on a high value target.
Sad fact is most of the companies that hold billions of dollars of customer assets do development exactly the same way a gaming company might. Those that even attempt security are unicorns. They bank everything on things like DNS, BGP, TLS certs, and ssh host keys working. This is like the medical industry before they learned hand washing was a thing.
I teach every one of my clients how to do things like commit signing, but for every one I help improve there are 100 more on track to be hacked any day now.
I can totally forgive a startup that cannot afford senior security engineers for a mistake like this, but Microsoft can afford that, or at least consultants, and yet they can not even enable optional signing for NPM, properly enforce git commit signing, or even protect an ssh host key trusted by millions in a TEE or HSM.
Any compromise would have to isolate them from the "real" Github hosts from today onwards, i.e. plausibly MITM them continuously, or they would just switch to the rotated key to be able to continue working. At least in OpenSSH, this means replacing the compromised trusted key, as there can only be one per host (or even IP in the default configuration).
This is still very bad, but much less catastrophic than e.g. a world in which `.ssh/known_hosts` allows multiple entries, in which case you'd really have a sustained compromise of most clients.
> UpdateHostKeys is enabled by default if the user has not overridden the default UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, otherwise UpdateHostKeys will be set to no.
This is an OpenSSH extension that allows any host to provide additional host keys the client will then silently add to `known_hosts`. This is really bad in this context as it can allow a one-time MITM to install rogue keys for github.com that can go unnoticed; check your `known_hosts` for any unexpected entries for github.com!
This is not hard. If it were hard, we wouldn't need encrypted connections.
If I were a nation state, it would be trivial to position an attack at the edge of a network peering point for a given service. How do I know? Our own government did it for 10+ years.
Cyber criminals often have the same tactics and skills and can find significant monetary reasons to assume heightened risk in order to pull off a compromise.
Random black hats enjoy the challenge of compromising high value targets and can find numerous ways to creatively infiltrate networks to perform additional attacks.
Even without gaining access to a network, virtually anyone on the internet can simply push a bad BGP config and capture traffic for an arbitrary target. Weird routing routinely happens that nobody can definitely say isn't such an attack.
You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance.
Tons of options. They clearly chose none of them.
Which ones?
> Also you can scale horizontally
For SSH? Only by having the same private key in all of them, which means it's still around somewhere.
> or only use the HSM as a CA to sign short lived host keys
That would be ideal, except that the user experience around SSH host certificates is currently woeful.
> Which ones?
Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway.
> For SSH? Only by having the same private key in all of them, which means it's still around somewhere.
End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works.
If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.
Genuinely asking because I’ve struggled with this question.
1. You compile a deterministic unikernel appliance-style linux kernel with a bare bones init system
2. You deploy it to a system that supports remote attestation like a nitro enclave.
3. It boots and generates a random ephemeral key
4. m-of-n engineers compile the image themselves, get the same hash, and verify the remote attestation proof confirming the system is running the bit-for-bit trusted image
5. m-of-n engineers encrypt and submit shamirs secret shares of the really important private key that needs protecting
6. key is reconstituted in memory of enclave and can start taking requests
7. Traffic goes up and autoscaling is triggered
8. New system boots with an identical account, role, and boot image to the first manually provisioned enclave
9. First enclave (with hot key) remotely attests the new enclave and obtains its ephemeral key (with help of an internet connected coordinator)
10. First enclave encrypts hot key to new autoscaled enclave
11. rinse/repeat
These kinds of changes suuuuuuck. Messing with known_hosts file is not always something easy to do. Might require a image rebuild, if you have access at all.
Do not put long lived cryptographic key material in the memory of an internet connected system. Ever. It is a really easy to understand rule.
- When exactly was the key leaked? Yesterday? A month ago?
- How long was it visible for? And no, "briefly" doesn't cut it.
- Did any internet traffic reach the page while it was exposed? We know you log everything, so it is a yes or no answer.
If any of these answers were pretty, I imagine they would have explicitly included them in the post.
> This week, we discovered...
So they found out about it sometime between Monday and Thursday, and rotated it Thursday evening.
Security issues are obviously very important, but equally I would be worried if they started acting impulsively immediately after discovering the issue, because there would not have been time to understand the problem and consider the options. If you jump on a solution without giving it a bit of thought first it's too easy to exacerbate the situation.
I don't know how long exactly, but in theory you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events
With these events you know what new repositories are created and what changes are pushed, so you can fetch every new change. Once something gets pushed to a public repository, it's very likely that some spider will have fetched it within a few minutes.
Wow I am shocked that they allow "firehose" access not only for free, but without even an API key.
Given enough disk and bandwidth, does this mean you could keep your own copy of all of github? I'd love to be able to grep the whole thing.
Are we talking 1TB/year, 10TB/year, 100TB/year?
"This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository"
> We have no reason to believe that the exposed key was abused and took this action out of an abundance of caution.
This is not verifiable, right? As the authentication method has no public and required revocation source, and given that the key, if having leaked, likely will be acquired by an authoritarian government, they can selectively MITM organizations and users that are not aware of this blog post.
(sorry, the comments are mangling this, clean version at https://gist.github.com/robbat2/b456f09b7799f4dafe24115095b8...)
``` # You might need to insert this in a slightly different place cat >>/etc/ssh/ssh_config <<EOF Host * RevokedHostKeys /etc/ssh/ssh_revoked_hosts EOF
cat >>/etc/ssh/ssh_revoked_hosts <<EOF # https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k... ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== EOF ```
Thanks for the gist though, seems helpful!
I expanded on your gist:
https://gist.github.com/e12e/0c1868479c0b8d0a52914d44be66d76...
$ ssh-keygen -lf github.old.pub
2048 SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 no comment (RSA)
https://web.archive.org/web/20230320230907/https://docs.gith...
(please check my work on your own machines and don't take my attestation on faith!)
RevokedHostKeys doesn't accept ~ for your home directory... while things like ControlPath will.
I'd rather confine this to my account, but I either have to use a relative path that doesn't always work... or a fully qualified path that includes my username (and may change)
> ... out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com
Yeah, that's not an "abundance of caution." That's the bare minimum response at that point. What's the "not cautious approach?" Make the repo private and go on your merry way?
I suppose "abundance of caution" might apply, if they determined that the only ways it could've leaked were from requests that were logged, and they've removed all the ways and checked all the logs.
But if I had to guess, even a brief exposure can be picked up by bots (and perhaps they can already see log entries for this). Even if no one at all picked it up, there'd still be the question of whether traces of it are still left behind on various infrastructure, in storage and caches (even ML training?) not intended for key safety.
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
[1] https://github.blog/changelog/2022-01-18-githubs-ssh-host-ke...
I looked at my `~/.ssh/known_hosts` file and that key is associated with a few IP addresses in addition to github.com. Those lines stayed after I ran `ssh-keygen -R github.com`.
I imagine that I also need to remove those other lines manually, but isn't that something that GitHub should have mentioned? I'm not sure in which circumstances these got added either…
- 192.30.253.112
- 140.82.114.3
|1|{base64-encoded-string?}|{another-base64-encoded} ssh-rsa AAA{long-string}
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
HTTPS has the same problem. Compromise any frontend, and you can MITM traffic to any other front-end till the expiry of the certificate (usually 90 days).
I would really like to see some kind of almost realtime delegation - so that for example, every second a new sub certificate could be generated and signed by a HSM, valid for only 2 seconds.
With HTTPS certs you usually have at most 90 days of impact when a key is leaked (less if you revoke and software is checking CRL). GitHub used the same RSA key for over a decade, they may have continued using this key for quite some time more had they not noticed the leak this week.
My immediate assumption wasn't that GitHub had changed keys. I thought that my computer was having some sort of problem. Only after searching around for a bit did I find out that yes, GitHub changed their private key.
Couldn't they have e-mailed all of their users and/or customers, to tell us what had happened?
• When you run a git command and git tells you about the mismatching key, it will prompt you to enter `y` or `n` but not actually let you do so.
• You need to run `plink github.com` to get the same prompt. You can then enter `y` to accept it.
• You should only accept it if it says the new fingerprint is d5:2c:63:d9:bc:75:9d:de:b1:4e:36:28:9f:7a:9c:39. If it is any different, you may be targeted by an attacker.
• The new key is stored in the file `C:\Users\<your name>\.ssh\known_hosts`. The new key at the time of this writing is `github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=`.
• In the future, if the key ever changes again, you can look it up on GitHub’s SSH key fingerprints page [0].
[0] https://docs.github.com/en/authentication/keeping-your-accou...
Seriously? How that happened is deeply concerning.
And why weren't the other keys exposed?
Most development work, however, uses SSH
The "insteadOf" git config is added to workstations and runners to convert attempted SSH connections over to HTTPS.
Like most corporate environments, “it is what it is” and we do our best to perform our jobs within these constraints.
But you've made me curious, what's your workflow like? Do you use a gui git client? For me, the default option provided to me by github is an ssh clone, at least for repos that I can own/can push to. This makes it very convenient to work with, because as long as I'm logged into a shell, I'm authenticated to github. But then again, I almost exclusively use the cli and have an ssh key configured for my gh user...
I personally never clone using https and I think most of the people I work with do the same
$ ssh -o VerifyHostKeyDNS=yes github.com
The authenticity of host 'github.com (140.82.121.3)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
No matching host key fingerprint found in DNS.
What a security nightmare…
# Add this to the top of ~/.ssh/config
RevokedHostKeys /home/username/.ssh/revoked_host_keys
cd ~/.ssh
mkdir revoked_host_keys.d
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' > revoked_host_keys.d/github-leak-2023.03.23
ssh-keygen -k -f revoked_host_keys revoked_host_keys.d/*
ssh-keygen -l -F github.com
[1] https://docs.github.com/en/authentication/keeping-your-accou...
[2] https://docs.github.com/en/authentication/keeping-your-accou...
Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.
I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
That's not "the whole point". You can have methods to copy a key between HSMs, methods that can scale just fine while being much more secure than a file in a repository.
I've never seen the inside of GH's network, but I would be surprised to discover they're not distributing the load of SSH termination across a fleet of machines. Just put a HSM on each of the machines that terminates a SSH connection.
It still shouldn’t have been easily accessible to anyone except the instances running the SSH service.
Yes, it makes things hard and unconventional to set up. But GitHub is not some small website.
Or passive! Probably it wasn't a real product but I recall reading about one that derived its key from the field generated by randomly arranged magnetic particles in the resin, or something like that. The point was to make it impossible to disturb the resin without altering the key.
Meta doesn't own Github, MS does.
When I tried to pull from my repo I got the warning message, right
I removed the old keys with ssh-keygen -R github.com
Then, trying with `ssh -T git@github.comp` I see this
The authenticity of host 'github.com (140.82.121.3)' can't be established. ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
So, first thing of course, is that the fingerprint does not match the one in the blogpost which is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s
Second problem, is that ip 140.82.121.3 seems to be reported as HIGH RISK[0]
So basically, how should I proceed? I am not security expert but I would say I am not an illiterate on this, and I have no idea. I guess the majority of users would you accept the new key, but is this the right move? I would need to do it if I want to do some work, that is for sure
EDIT: Formatting
[0]https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-test...
Not in the blog post but the blog post points to my link which is the official documentation.
1. https://docs.github.com/en/authentication/keeping-your-accou...
I manually added the new RSA SSH public key entry to my known_hosts file (like they say in the blogpost)
Then ran ssh -T git@github.com and got
Warning: Permanently added the RSA host key for IP address '140.82.121.3' to the list of known hosts. Hi gassius! You've successfully authenticated, but GitHub does not provide shell access.
Then, when trying a git pull, I got this:
Warning: the RSA host key for 'github.com' differs from the key for the IP address '140.82.121.4' Offending key for IP in ~/.ssh/known_hosts:63 Matching host key in ~/.ssh/known_hosts:64
So basically, the Offending key is the one I added manually as per blogpost?
Ok, I am in Europe, and this seems like an issue of global distribution network or something, but this is not great AT ALL, either the blogpost information is not complete or something fishy is going on
UPDATE: The replies makes clear what I was seeing those errors and make sense. Thanks
EDIT: Formatting and Acknowledge of the situation per replies
I get this error each time I interact with Github: The authenticity of host 'github.com (140.82.112.3)' can't be established. ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
When I type 'yes' I get this added to my host file: 楧桴扵挮浯ㄬ〴㠮⸲ㄱ⸲″捥獤ⵡ桳㉡渭獩灴㔲‶䅁䅁㉅橖䡚桎塌潎呙瑉浢穬䡤祁呎䅙䅁䥁浢穬䡤祁呎䅙䅁䉂䕂䭭䕓橎䕑穥浏歸䵚㝹灯杋䙷㥂歮㕴剙奲橍畎㕇㡎男杒㙧䱃扲㕯䅷呤礯瘶洰噋唰眲地㉚䉙⬯含潰正㵧
ssh-keygen -R github.com
ssh-keygen -R 140.82.121.3
Warning: the RSA host key for 'github.com' differs from the key for the IP address '140.82.121.3'
Offending key for IP in /home/tom/.ssh/known_hosts:77
Matching host key in /home/tom/.ssh/known_hosts:102 sed -i.github-removed '/AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa/d' ~/.ssh/known_hosts> GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
What the... That's not "an abundance of caution". That's the only possible course of action.
Do they expect people to think "Wow, Github leaked a key, but even without knowing if anyone snagged it, they're still replacing it. Wow, they go above and beyond."
It's so ridiculous.
After all it’s them hosting and serving the requests for that (and every other) repo.
Only _reasonable_ course of action. Possible is to do nothing =)
That's the "caution" part.
https://docs.github.com/en/authentication/connecting-to-gith...
(it would actually be best to update it to the Ed25519 public key, as it's the recommended algorithm when generating a new SSH key).
https://lwn.net/Articles/637156/
I'm honestly not familiar with anyone actually using host-key rotation?
I mean it seems like its clearly a key that wasn't in an HSM.. and over the lifetime, hundreds? Thousands of Github employees could of accessed it?
Anyone who's baked the host key in the known_hosts file that gets shipped on their CI systems would start to see jobs failing, and have to manually fix it up with the new host key.
These things are just annoying enough that I think it's perfectly understandable that GH doesn't want to regularly rotate this private key.
The current rotation allows anyone to try to fish the lazy users (like me probably) who will just trust on first use. Probably a bigger risk than key compromise, since they have logs.
It could be a better idea to use Host-key rotation, despite it making the life of a key-thief a bit easier. Just because it exposes people less against opportunistic impersonators.
2. It was only added in OpenSSH 6.8, so it missed Ubuntu 14.04 release, and only really turned up in 16.04 LTS that way, plenty of old systems it wouldn't work on.
As other posters noted, a bad actor could rotate the key to their chosen keys just as easily as GitHub could cause the rotation.
It seems like at least a `known_hosts` compromise would be "self-healing" after connecting to the legitimate github.com server once.
=====
After I followed the instructions to remove the old RSA key, `git pull` started using the ECDSA key, and now shows this warning:
Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '20.205.243.166'
Offending key for IP in /home/forge/.ssh/known_hosts:87
Matching host key in /home/forge/.ssh/known_hosts:88
Are you sure you want to continue connecting (yes/no)?
ssh-keygen -f ~/.ssh/known_hosts -R 20.205.243.166
I'm not sure why the ECDSA key of the github.com IP has supposedly changed - maybe someone can clarify this?
Since GitHub's IP address is not stable, I suggest disabling the IP-checking feature using
Host github.com
CheckHostIP no
It would be so much better if standard practice was to generate and store the private key on a smartcard or the TPM, so that the only file a clueless/careless developer could upload would be a stub.
Yep. Especially given that basically all modern laptops (and some PCs) ship with TPMs and ssh can use it via the TPM PKCS#11 lib. I'm using that daily on multiple machines and it's working great.
It really should be named `id_rsa.private` to help a busy developer realize they have the wrong file.
It wasn't even that long ago that Github changed the checksums of git tarballs which also broke builds everywhere. All we got was a "oh btw we changed the archive behavior" blog post after the change went live [0].
[0] https://github.blog/changelog/2023-01-30-git-archive-checksu...
I'm upset they haven't sent out an email to every user. This affects their entire usebase and basically everyone needs to take manual action to continue working. If you haven't seen the news then you'll see failed builds/git commands.
At least put your key to https://github.com/.well-known/ssh/ed25519.pub so I don't need to Google it... And may be some day ssh will support it natively. Someone need to act first.
Here's the list of keys if anyone needs:
https://docs.github.com/en/authentication/keeping-your-accou...
> Host key for github.com has changed and you have requested strict checking.
> Host key verification failed.
It is also not mentioned on https://www.githubstatus.com/ aka status.github.com.
curl -s https://api.github.com/meta | jq -r '.ssh_keys | join("\n")' | sed 's/^/github.com /' > /etc/github_known_hosts
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/etc/github_known_hosts" git pull
$ openssl s_client -showcerts -connect github.blog:443
CONNECTED(00000006)
depth=2 C = US, O = Internet Security Research Group, CN =
ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = github.blog
verify return:1
---
Certificate chain
0 s:CN = github.blog
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
v:NotBefore: Feb 3 17:19:36 2023 GMT; NotAfter: May 4
17:19:35 2023 GMT
... ssh-keygen -R github.com && curl 'https://api.github.com/meta' | jq -r '"github.com " + .ssh_keys[]' >> ~/.ssh/known_hostsAlso you should first check:
1. If you’re using GitHub’s rsa key at all
2. That you don’t have the other keys
In which case nuking all the keys is completely unnecessary, just remove the RSA one.
But how do you know it's the right key?
Funny enough, given that every person (and every bit of tooling) I know blindly approves SSH public key verification anyways, they will likely not even notice this switch.
Maybe on initial connect, but who TF ignores that "key has changed danger danger, high voltage" warning? You at least look around and ask a colleague?
If you don't pre-bake the known-hosts, then you'd allow each new ephemeral run to use whatever github tells you it's key is.
It did briefly break ours, as we pre-bake the known_hosts file into our CI image for convenience and security.
Convenience due to CI not having TTY's, so various tools would get stuck on prompt Y when connecting to github for the first time. (Which is every run, if you are ephemeral CI)
And security, as now everything broke due to github's key changing, which is the desired outcome actually.
We bumped the known_host key entry, merged and all is well again...
Given that the key is extremely long lived, this has unfortunate implications: If any of these servers is compromised, or decides to go randomly spewing memory content because of a bitflip, or screws up the nonce on a DSA/ECDSA operation, the key can be compromised. This is hard to exploit if you're a random person, but for a global adversary that collects internet traffic at scale (e.g. NSA), it's feasible and I would be surprised if they weren't exploiting such issues. They were collecting HTTPS handshakes for a reason.
You can't get two (well functioning) TPMs to have the same key. They come with their own, un-extractable* and unchangeable keys built in.
*TPMs claim this, it is probably not impossible to extract keys just incredibly difficult and requiring specialist knowledge.
All cloud vendors offer the same functionality, if you think about it, so it's not an obscure feature.
If anyone is using a Windows machine with TPM-based bitlocker encryption, you have followed the instructions at https://support.microsoft.com/en-us/windows/back-up-your-bit... I hope?
1. Various DNS hijacking and cache poisoning attacks 2. three letter agencies in meet me rooms 3. Exploited/hacked routers 4. Wifi hot spots
Users will need to actively remove the old rsa key in order to be safe. It's my first question, and a colleague suggested that they believe that the private key was not seen, however, I don't see that in the post - unless I'm missing it - and I really want this stated very clearly somewhere.
Yea that is my read on it as well. If that is true this is much more severe than the blog post suggests.
To be honest, I'd expect something like this to be mentioned in the announcement.
If they did something that expensive for security they would be bragging about it from their blog to earn any possible trust they could at a time when they could really use it.
If you are reading this Microsoft, make me look silly. Please.
I will even give you the enclave designs for free.
ssh-keygen -f ~/.ssh/known_hosts -R github.com
Note that github-keygen stores the ssh-ed25519 key since release 1.306 in June 2022. You can check by yourself:
cat ~/.ssh/known_hosts_github
curl --silent https://raw.githubusercontent.com/dolmen/github-keygen/release/github-keygen | perl
- An SSH key can be revoked on the server, but the client won't register it immediately, and all clients must manually verify the new key signature and update their local configuration. This doesn't happen with HTTPS; the client just works with the server's new cert, no user intervention is required.
- A username can be used in the HTTPS URI to tell both your client and the server what credential to use. SSH method requires the user to load the correct SSH key first.
- Most servers like GitHub allow more fine grained access control for HTTPS tokens than SSH keys.
- HTTPS access method works on all HTTP proxies, whereas SSH is often blocked.
- HTTPS can be faster than SSH.
- HTTPS allows a client and server to use specific ciphers to address regulatory and other requirements.
- HTTPS tokens are supported in a wider range of password managers/keychains than SSH keys.
- Most users don't password protect their SSH keys, but they often have a password manager with a master password that they can keep HTTPS token in.
- SSH key management is much more complex on the user end than HTTPS token management. The former requires ssh-agent, a key generator, and instructions for use, as well as specific filesystem permissions for keys.
- The user often doesn't understand the idea that they shouldn't share their private key. But everyone basically gets they shouldn't share their password.
- Virtually no one ever checks the fingerprint of ssh server keys. Often entire companies have configuration that disables host key checking, completely eliminating the security SSH is supposed to provide. With HTTPS the user is implicitly protected with no actions necessary.
- Nobody ever commits a private TLS key to a GitHub repo, but apparently they do with SSH private keys...
Oh how I wish this were true...
Are there password managers that actually seamlessly do that? GitHub's official docs (https://docs.github.com/en/authentication/keeping-your-accou...) for example recommends using the GitHub CLI to cache your authentication token for you and as far as I can tell the token is stored unencrypted (https://github.com/cli/cli/issues/1773), unlike SSH keys which should be encrypted behind a password. If you don't do this you have to enter your password every single time you use you Git, which is not good even if you have a password manager (it adds steps, and passwords aren't great to begin with because you send them in plain text to the server and it is why Apple etc are moving away from password-based authentication for websites).
> - Nobody ever commits a private TLS key to a GitHub repo, but apparently they do with SSH private keys...
I think the key issue here is this really shouldn't have happened. There is no reason why GitHub couldn't secure their SSH private key just like their TLS key.
---
Edit: Thinking more about it, I think the situation would have been much worse if GitHub leaked a TLS key instead.
With SSH, an attacker who stole GitHub's private SSH key actually *cannot* MITM the user. They can pretend to be GitHub to the user, but they cannot pretend to be the user to GitHub, because with SSH both sides have a private/public key known to the other side. In a normal usage scenario, it would actually take some social engineering to do a real attack, because a simple Git fetch from the attacker would fail when trying to fetch a private repo (since the attacker does not have the permission to fetch it from GitHub).
With HTTPS, under current setups of username/password or an auth token, the user has no real way to protect themselves from a MITM attack. The token isn't a private key as it's shared directly to the server. That means the attack could actually MITM the server/user connection and do much much more damage.
Also, SSHFP requires DNSSEC, which Github of course does not support. (amongst other shiny new technologies, such as IPv6... /r).
And even if GH would deploy DNSSEC, it'll still be opening you up to a host of other attack vectors that come with DNS based trust anchors.
That's bad advice. Everyone should remove the compromised key, not only people seeing the message. If you don't see the message and everything still works (while using RSA), you've been MitMed.
error pulling image configuration: download failed after attempts=6: x509: certificate is valid for *.githubassets.com, githubassets.com, not pkg-containers.githubusercontent.comLet's remove the words briefly exposed and public. Why is a private key in a Git repository? These days I don't see any reason for this.
Very opportune time for hackers to try this, since few are probably verifying the fingerprint.
How is this true? AFAIK, A MITM with the github priv key will be able to do an SSH downgrade attack...
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
No further questions for this witness Your Honor
Something like that must have happened anyway, since it's highly unlikely a private key is just lying around as a plain text file on an engineers workstation to be accidentally included in a push.
> Could anyone with that key have cloned any GitHub repo, private or not?
If you try to push to GitHub but someone is MITMing you with the leaked key, they can say "I'm an empty repo" and the client will helpfully push all the commits to the attacker.
Until everyone updates their .ssh/known_hosts file, a MITM attack can still steal code in this way.
They seem to have been using the same key.
- Push security breach details down so fewer people read them. Nothing until line 11. Or line 13 if you count the TLDR.
- Litter your text with understatements. After publicly exposing keys, you replace them "out of an abundance of caution" to replace them.
And yet this is one of the best security breach posts I've seen in a while. When will companies start to respect developers?
Do the PR people have technical knowledge or does some tech person write
“I accidentally screwed up and put the key public so we have to replace” and the PR glosses over it and rewrites “Out of an abundance of caution…”?
Hyperlink with broken certificates. Fun.
I know I know. All you have to do is to just MITM someone's DNS or IP traffic or whatever and it's all over. Well, I will guarantee you that millions of bitcoin are sitting there waiting for you to take it. It's all yours. Go take it. You could be a millionaire tomorrow. And if you do it right, they'll let you keep some of it if you send most of it back.
I'd love to get a copy of the key, so I can setup a demo to encourage education.
> That said, if I'm honest I think almost no one checks the SSH host key. I would bet that for 9 out of 10 engineers / devops / security people, they will ignore the SSH Host Key and might even set StrictHostKeyChecking=no.
I believe you, and I agree this is common. I have been carrying the github host keys in my dot files for more than a decade. I do this somewhat ergonomically by having a Host config that adjusts the known_hosts path to permanent_known_hosts for things I care about.
Setting StrictHostKeyChecking=no is negligence, despite being common there really isn't two ways about that, it's a trivial vector with only a very moderate need for additional information to turn this into an RCE in most cases.
There are big problems with ssh's host keys, I understand and agree, but there are also big problems with passwords and we've made a dent in getting people to use password managers and 2FA. We should not wave this away just because it's a little harder, otherwise we're just relegating ssh to the same status as gpg.
Edit: where can I get good Mercurial hosting? F this.