Netgear Signed TLS Cert Private Key Disclosure

It should be noted that it’s essentially never the people who find bugs arguing for “responsible disclosure”.

It sure is easy to tell others what to do with their work product when you have zero stake in the game.

There exists a really easy solution to the purported problem of full disclosure, vendors could just offer significant enough financial compensation for non-disclosure.

Because this post is specifically a found private key for a certificate in the Web PKI it was not necessary to post that key in order to achieve all the positive consequences of public disclosure. The key only enables negative consequences.

I could give some leeway to a grey hat who finds the data but doesn't understand what it is and posts it. "Hey, what's this blob of data?". But this poster clearly understands it's a private key for a certificate in the Web PKI.

You can disclose the fact that you know a private key that isn't yours without revealing the key by various methods, but one that's very easy for non-experts is to create a bogus CSR. Just tell OpenSSL (or a tool that's actually good) that you have this private key and you want to request a certificate for it. Give bogus details, for example in the Common Name of the proposed certificate you can explain this is a Netgear key you found.

You can now publish the CSR, it is inherently proof that you've got Netgear's private key but it does not contain that key and so black hats will need to do their own work, for which you are certainly not responsible, to get that key from a Netgear box if they want to.

It would also make sense to tell the issuing CA, you should send them that CSR, although it's not terribly harmful to send them the actual private key and I guess if you're worried the CA's representatives don't "get it" this is a very blunt way to make your point. If the issuing CA doens't respond, tell m.d.s.policy both that you found this key and that the CA did not respond, and if necessary that can be escalated until the CA is distrusted (by Mozilla, and in my experience eventually everybody for reasons we'll not think too hard about here).

In the case of Let's Encrypt the process is, like everything else, fully automated. Call the API (by running your client software of choice) and prove you know the private key for a certificate they issued and want it revoked, its status changes to revoked and the next batch of OCSP signatures will show revoked for that certificate.

Agreed, the key was already publicly available, and it was just a matter of time for someone with malicious intent to find it if that's not already the case.

Exposing something already public to speed up the resolution and make sure the impact is kept to a minimum considering the circumstances.

I have posted this in another thread, but I am posting it again because more people need to be aware of Netgear's practices:

I recently bought a couple of Netgear Managed Switches (for Business)⁰ and in their datasheet they list "Local-only management" as a feature. Only after they arrived we discovered that you only get limited functionality in the Local-only management mode, you have to register the switches to your Netgear Cloud account to get access to the full functionality.

Reading up on it, this was achieved only after a community outcry because in the prior firmware versions the switch would have to connect to the Netgear Cloud on every bootup.

Needless to say I would not have bought the swiches if I had knew I needed to register them to Netgear Cloud to have access to the full functionality specified in the data sheet. If I had bought them as a consumer, not as a business, I would have returned them immediately.

Netgear are now on our purchasing blacklist.

⓪ - the switches are Netgear GS-108Tv3

True, but the devices themselves are often pretty powerful in the home networking space, especially for the price.

I've got an R7800 running as the main router in my home, however, it's flashed with OpenWRT as their own software is a complete mess.

Bugcrowd run bug bounty programs like https://bugcrowd.com/netgear on behalf of companies

They explained earlier that if they entered the bug bounty program with Netgear, they would be agreeing to not disclose: "By submitting the security bug, you affirm that you have not disclosed and agree that you will not disclose the security bug to anyone other than NETGEAR."

So it seems they were trying to get Bugcrowd as a intermediary to contact Netgear on their behalf to "establish a communications channel outside of the Netgear bug bounty programs"

The difference is that these newly discovered certs with private keys were signed by real Certificate Authorities trusted by browsers by default.

When I can control the client configuration, I just set up my own trusted root CA and then have the distributed systems automatically submit a CSR on initialization. This doesn't work when you have to support end user's web browsers though (unless you're somehow lucky enough to own a public CA and don't care about ICANN rules). It really does seem like there should be an encryption-only option for TLS. Maybe anything that resolves by IP address and not name should display the green padlock even if the SAN doesn't match the name?

Unfortunately this is a problem that's mostly a result of the browser vendors (who are in control of the validation/UI), and can't be fixed by Netgear / etc. They've decided that security trumps usability.

I suspect at done point the router manufacturers will give up and just force setup through a phone app (assuming you could download it over a cellular connection). That makes me sad, though.

"How" is that they only inlude the private key for that specific router, and then that key/cert itself is certified by the real CA cert, the key for which is not kept on the router itself?

1. Just let it be HTTP. Stupid browsers are stupid, but at least they don't prevent this page from working yet.

2. Router coordinates with company server to get its own hostname like n-123123123.netgear.com (which points to 192.168.1.1 or whatever), generates private key and company server issues certificate for that key. HTTP requests to 192.168.1.1 return HTTP 302 to this address. It requires Netgear to operate CA or make some extended agreement with existing CA to let them issue certificates. I know that Plex does something similar, so it should be possible.

3. Company server creates hostname which points to a public router IP address and router uses letsencrypt to get a certificate for that hostname. But that requires public IP and some providers are using NAT nowadays, so it won't work universally.

4. To configure a device you're talking with company server, rather than your device and your device is talking with that server as well. It requires working Internet, so not a complete solution as well.

I don't believe that HSM chip would work. You would just extract that chip and use it as a private key and that's about it. You won't be able to extract private key bits, but you don't need them. May be it might work with very secure device like iPhone, where everything is signed, but yeah, $10 device and someone will still hack it, causing bad situation.

We could change the standards to facilitate client apps using a private network's own PKI. This would solve about a million problems that currently exist with most private orgs that need to enforce policy on content on secure networks, as well as allowing consumers to securely browse self-hosted private services. I don't know what that would look like or what would need to change, but it could definitely be done.

Ideas:

1) An extension to PKI that browsers support. If you browse a service with a certain type of cert on a certain TLD, a) it won't work except on private networks, b) it will prompt the user to enter a key, such as a key printed on the backside of a router (this is already used to get consumers onto WPA routers, so we know it's reasonable). The device with the cert creates the cert itself. If the key is right, you can browse the service, and your browser caches the key with the cert. If an attacker learns the key they can try to make a new fake cert, but the browser would know it's not the same as the old cert and block it. Replacing these should be quite infrequent, as often as people replace their home network devices.

2) A TLD that browsers will only ever respond to if its names resolve to private address ranges. If a service wants to serve HTTPS privately, it will request a cert be signed by a device that is auto-detected on the local network. The device will return signed certs for a particular host on the local-only TLD. The device can have any logic it wants to restrict what can get a cert and how. The trick is how to make the client trust the device, which I think 1) would solve. This device can, for example, only ever create a given FQDN's cert once; if an attacker tries to generate a duplicate, it will fail. If the cert ever needs to be refreshed, the memory/flash of the device needs to be reset. This makes it possible to have a captive portal request a cert from a local router, and no attacker can generate an identical cert unless the router was reset and they triggered the cert generation before the captive portal did. (Also for captive portals, the admin of the router could just hard-code what clients could request what certs, so the captive portal could always request particular local certs for itself, while no attacker could). It's still tricky to determine trust on first connection in the case of things like coffee shops, but maybe you could receive a public key via WPA or something (then again that's kinda kicking the can down the road to WPA)

Ubiquiti do this by tunnelling it through their servers. You can still get insecure access by going direct to the IP, but going to the Unifi site gives you full HTTPS.

Plex do it with a wildcard and generate a new one for each server: https://blog.filippo.io/how-plex-is-doing-https-for-all-its-...

The other way this could be done is if TLS-SRP was more widely supported. This variant of TLS doesn't depend on certificates at all and instead uses mutual knowledge of a password to authenticate the connection in both directions. In the case of a router this would be written on a sticker on the device for initial configuration.

It’s impossible, they need to try something else. They could get dedicated domain names for each router, for instance.

No, you can do something like CloudFlare does for TLS for their enterprise customers, where the TLS session is signed by a key held by Netgear.

https://www.cloudflare.com/ssl/keyless-ssl

The fallback would have to be self-signed TLS with random keys for initial local router configuration, which is hopefully done over an Ethernet connection rather than WiFi.

You don't need a secure enclave or HSM to solve this problem. It just requires caring about security and provisioning enough time to engineer and test the obvious solution. From what I can tell, that doesn't line up with Netgear's MO.

I think in this case it's to force browser vendors (who have the most exploitable endpoints) and companies like Apple and Microsoft at the OS level, to blacklist the offending certs.

Though the CAs in question should probably have an automated challenge-response system to which a timed signed reply of a given message of their choice causes a revocation of the key that signed the message. (As sufficient proof of "this is vuln, kill it now, ask questions after".)

> 6 days is nowhere near a justifiable timeframe for full disclosure.

Of course it is, if you didn't get a response in the first 48 hours you're not going to, that's the way things like this tend to operate. The security contact for all companies is either a) inactive or b) very active. It's a detriment to everybody that for some reason it has been normalized that people should wait months or years for disclosing bugs. Full disclosure is the only way things get fixed.

> This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP.

Github is archived in real time for the most part. Especially you'll notice that if you post a private key for a cryptocurrency address or credentials for AWS, it'll be stolen and used within seconds. There's some really good sets of information out there like https://www.gharchive.org/ which give you an idea of the sheet amount of data that github produces on a daily basis, and that's just the metadata and things like comments rather than actual git repository contents.

The idea that you could delete something from there and have it actually "removed from the internet" is amusing.

There is another reason why responsible disclosure would have been better (and less short-sighted): vendor might make an internal audit, find root cause and invalidate a bunch of certificates at once (think about similarly leaked certificates due to a bug in some deployment tool).

Now it's an open race with the bad guys, and surely a lot of them all at once; hardly an advantageous scenario for end users.

Responsible disclosure is meant as a way to harm researchers and users for the benefit of companies. This is absolutely the right thing to do as a security researcher, especially given that Netgear prevents public disclosure, ever, if you submit a bug bounty. To make sure it stays online:

https://web.archive.org/web/*/https://gist.github.com/nstark...

“ Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs.”

I’m curious to what does this mean? They got a response form Bugcrowd but not Netgear themselves? Or they got through to Netgear but didn’t like what Netgear said so they went public? Something else?

I discovered this same thing in 2017. My bugcrowd report was shot down as dupe

https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-rout...

Full disclosure means 0 days generally.

Boo hoo. Netgear should have known better than to expose their keys like this in the first place, this is really amateur hour stuff.

They were reached out to and didn’t respond. It’s their own fault and they have to make do with what they’ve been granted. Now they get to pick up the pieces.

Full disclosure is always perfectly justifiable, unless you’re stealing someone else’s work.

It wasn't the researcher who disclosed the keys, it was the vendor who did, knowingly and purposefully (in their firmware updates).

Who cares, this does not even warrant a responsible disclosure. Netgear can't fix that in a way that makes everyone happy. You can revoke the cert, then what? A firmware update with a new cert? Same problem, private key will still be on the device and in the firmware file. Self signed? You get users crying about the cert. HTTP only? You get users crying about "not secure" even though that might not be true in all cases.

My guess is that people before the ones in the gist have tried to contact Netgear about this (since it's not hard to find) and chose to ignore it.

I think https could work fine, just give each device a unique key and dns name.

> Non-technical users get an app ...

So now I need an already-working smartphone to set up my internet connection.

This could be extended to something like:

Ship a USB stick with the device with installer software on it.

The installer software sets up an SSH tunnel to the device, so it includes an SSH client and a script (batch) file that effectively does: SSH -Llocalhost:<some random local port>:localhost:<installer initial port> setup@<ip address of device>

It has the known SSH signature of the device in its known_hosts, and of course the user has to type the password from the sticker on the device.

Then it fires up the browser with the url http://localhost:<local port used in previous step> which has access to the full setup of the device via the browser. It is HTTP, therefore there are no initial setup TLS issues with the browser, as you are going via an encrypted tunnel set up via SSH using HTTP encapsulated within that encrypted tunnel (effectively a temporary VPN) rather than using TLS.

As part of the setup process, the device generates a self-signed cert, which you can put in your computers/browsers trusted cert store for future use. Also as part of the setup, once the self-signed keys and certs are generated, it disables this local HTTP webserver that is being forwarded to via SSH, so in future the user can use a standard TLS connection to the device using the generated certificates. If a factory reset of the device is done, it blats its config and re-enables the 'setup' webserver (that is only running on localhost therefore can only be access locally,i.e. via SSH tunneling).

Anyone with access to the private key can submit a problem report to the CA, they're obligated to revoke if the key is exposed to a non-subscriber.

I have submitted a report - they are obligated to revoke within 24 hours.

Chinese gaming VPN service or something, bypasses China IP blocks to allow them to play on NA/EU servers.

See DNS CAA https://en.wikipedia.org/wiki/DNS_Certification_Authority_Au...

mini-app.funjsq.com is revoked (https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b...), can't check routerlogin.net/com as they didn't paste the public cert.

On local networks TLS doesn't make things more secure, maybe even less in some scenarios given all the obtaining of certificates over public internet and leaking info to the public internet. So sane thing to do would be plain http. But really, browsers should stop the bullshit with https "security" and accept ssh-like behavior at least for local networks.

Define "secure". If you've "stuffed the key in there" in such a way that other people can get it out again, and it works across all routers of that model, then it's possible for an active attacker on your network to MITM your connection to your router. So it's slightly more secure than cleartext but not much; about the same as a random self-signed certificate.

It is definitely a hard problem because there's no easy way to authenticate the router to the client, but on the other hand I'm not sure how important all of this is when it's on a local link anyway.

A "best possible practice" solution would be to have the routers issued with individual certificates at factory programming time, and provide a rollover mechanism through the admin UI.

This was answered on the Gist in a comment[0] linking to a tweet[1]:

Things to do instead of shipping TLS certs and static private keys to consumer-grade routers you sell by the thousand:

Generate a unique keypair per device.

Use this keypair to communicate upstream, in a similar fashion to CloudFlare's Keyless SSL.

This keypair you generate on the device would need to be preloaded at the factory, unique per device, and the public key would need to be stored in a database.

When a TLS handshake comes in, you verify the entire request is signed by that keypair before signing it.

Delegated online signing for a TLS handshake isn't even in the top 10 engineering challenges for cryptography in the 2020s.

Oh, and, if you don't have connectivity to upstream and your need users to connect to configure something?

Fallback to HTTP reachable via port 80 on the private IP for the server.

[0] https://gist.github.com/nstarke/a611a19aab433555e91c656fe1f0...

[1] https://twitter.com/CiPHPerCoder/status/1219228544548720640

This. Yes, Netgear has a bad track record with security, but this is likely a calculated trade-off.

Alternatives:

- Self signed certificates. Users will learn to acknowledge the error and an attacker can just present their own self-signed cert. This what most vendors do.

- Plain HTTP. Worst option, no security, not even against a passive listener (but no bad PR because of "security disclosures"... hooray!).

- Something similar to Keyless SSL, as suggested on GitHub and in this thread. Will at best slightly inconvenience the attacker, who can still extract any device key and impersonate the device. It also introduces a single point of failure. Server is down, nobody can log into their devices (and HN would not be happy).

- Secure enclaves and hardware key management for the shared key. This would be workable and reasonably secure, but the implementation costs likely won't fit the threat model.

- Issue an individual SSL cert for each domain and print the domain on the box. That would work, but be very complex to manage (plus certificate costs, and plenty of failure cases around renewal).

- Reverse proxy through vendor servers. Single point of failure, and the most likely time to log into your router is the internet is down.

Hardcoding a valid SSL cert for a single-purpose domain (routerlogin.net) is similar to a self-signed cert in terms of security, provides a better user experience and does not teach them to blindly acknowledge TLS errors. The threat model requires an attacker doing a MitM attack on the local link. Also, compromising a router provides very little access to an attacker in today's TLS-by-default world.

Anyone downvoting the parent post should offer a better solution.