sullivanmatt

Ubuntu 22.04 & RHEL 9 are the major distros impacted. Docker images built on ubuntu:latest will also be impacted. The latest releases of Alpine/Debian/AL2 are all not impacted, they use 1.1.x lineage.

The email address listed in the blog post, securitykeys@cloudflare.com, appears to not be a valid mailbox.

Not a complete nothing burger; a lot of people here work for companies that sell to the Feds or host FedRAMP-authorized SaaS solutions. There will definitely be private-sector impact from that risk framework, though I'm not saying that's necessarily a good or a bad thing.

You have a very neat website (https://sam.design). Learned something new today.

I'm a security professional with a decade and a half of hands-on, real world experience. My most recent position being the product manager for Identity and Access Management for a leading B2B SaaS, dealing with real world attacks from extremely sophisticated threat actors. I assure you I know what I'm talking about, I've lived and breathed it every day for many years and have followed these standards since their initial drafts.

If you have knowledge on superior ways to protect users from MFA passthrough, please share it. I am always happy to learn about better ways of doing things. But contrarian bandwagoning without providing effective alternatives isn't helpful.

[removed by author]

A lot of people still have legacy Yubikeys floating around, and these are replayable. What you need now is something like the Google Titan FIDO2 key or one of the Yubikey FIDO2 keys. Transitioning an entire company to these, getting everyone to self-enroll, and then removing the ability to use all the less safe options across the employee base, contractors, etc is not cheap nor easy, and of course requires a massive amount of retraining. It's not as trivial as just buying them, sadly.

I think it's worth repeating: at this point, MFA that is not based on Webauthn (https://webauthn.guide/#about-webauthn) should be considered dangerously insecure. Uber almost certainly enforces MFA for remote access; I strongly suspect we'll end up hearing that it was successfully provided during the authentication step (update: screenshots on Twitter appear to confirm this). As we saw in the case of the 0ktapus campaign, a sufficiently-skilled attacker will simply proxy the MFA calls to the real identity provider in real-time, the user none the wiser.

Webauthn, however, binds the authenticator to the domain and port, and requires https as the scheme. If a user gets phished, they cannot be compromised: the phisher's domain will not match and any Webauthn authentication challenge would fail.

So if your workplace is letting you authenticate with SMS codes, push notifications to an app, or 6-digit codes generated by an authenticator app/hardware device, you need to start banging on pots and pans up your reporting chain to get your security team the support they need to make Webauthn + FIDO2 hardware tokens or Webauthn + Mac Touch ID happen.

In this case yes, it sounds like the pessimistic viewpoint may be the right one. However, for a while now HN has been going downhill with armchair analysis that inevitably concludes that any action taken by a business is wrong/harmful/bad, so I am thankful you took the time to be thoughtful before reacting.

Thanks Matt, not sure how I missed that. Glad to hear it!

For any of the Cloudflare team that frequents HN, curious if you have an eventual plan to open-source Pingora? I recognize it may stay proprietary if you consider it to be a differentiator and competitive advantage, but this blog post almost has a tone of "introducing this new technology!" as if it's in the cards for the future.

I do wish this wasn't a sponsored ad masquerading as an engineering blog until the last paragraph, but yes, these things are extremely challenging. What people miss is that the big cost isn't building or integrating these things, it's supporting them. I was the IAM features product manager for a mid-market B2B SaaS and had to jump on calls multiple times a week to help our Fortune 500 client base with planning our their deployment of directory integration (SCIM). You would think you could train support staff for the task, but with each IdP having its own zany behaviors and nuances (or just not following the protocols at all, cough Azure AD), you end up needing to pay someone who is extraordinarily technical to be an absolute authority in how these things work and are managed at both ends of the connection. I was backstopped by two other support engineers who could help with the day-to-day and escalated to me as needed. So multiple staff level engineers, all just to support two identity related features (SSO, SCIM) for one mid-market SaaS.

Everyone likes to gripe (including here in this thread!) about B2B charging for Enterprise features like SSO, but withhold your judgement until you've actually lived through supporting it. The burden of helping a company like Meta integrate their weird, bespoke, homegrown identity solution into your business app could literally cost your business tens of thousands of dollars per year (and no, that's not a made up example).

I also live in Ames, Iowa, where the author resides. I live less than a mile from the track segment he is talking about, owned and operated by Union Pacific. While I'm pretty skeptical of this author's individual concerns given the twice-removed nature of the statements from their "source", trains have unmistakably gotten longer during my 15 years near these tracks, and they are moving faster.

A hazmat situation impacting only a one-mile radius around the tracks could effect up to 30 to 40 thousand people here. I interned for Union Pacific while in college, and yes, the company will state that safety is their number one priority, but it was very clear that their business was to profit by optimizing every single little part of their business for maximum efficiency. The company has an open distain for regulations, especially safety-related regulations, and did not shy away from sharing those views internally. I would not be at all surprised if crews are being abused, and safety sidelined, for the sake of maximizing profit.

Mirror

https://web.archive.org/web/20220223141210/https://mwi.usma....

Personally, I already pay for Google One, run my phone lines on Google Fi, and only use YouTube for an average of 15-20 minutes of content per day. Google Fi aside, surely somebody, somewhere, at Google can just bundle these things up and I can just pay a single monthly rate to be "in" the Google universe? I'm not paying $12/mo for something I barely use, to a company that I already give money to for a "premium" membership tier. I also don't have the time to check what crazy bundles Google is attempting to offer this month that will simply not exist next month. I am happy to pay for Google's services, but why do they feel the need to make that an actively hostile process?

There's an entire market for this (and I own two for this express purpose): https://www.etsy.com/listing/1015753561/the-original-apple-a...

I'm working on getting the documentation written for it, but Atom feeds per category are recently live! https://bugalert.org/feeds/software-frameworks-libraries-and... https://bugalert.org/feeds/test.atom.xml etc

Category list is found at: https://github.com/BugAlertDotOrg/bugalert#category

Again, working on getting this all up on the site soon if you'd prefer to just check back later.

I've coordinated an NCC crypto audit, and I can assure you that is not the case.

Pinterest voluntarily entered this agreement, which provides better pricing in exchange for committed use - an offer AWS provides to many of its customers. This is a highly sensationalized headline.

Mongo adoption has risen at a rapid rate, and cloud services now make up nearly 40% of their revenue (which, in turn, allows the company to push those $$$ into the core product). Your statement, while a common view on HN, is not correct. Just because tech bros have moved on towards other interests has not meant that adoption of Mongo has faltered. (I am not affiliated with, nor own stock in, MongoDB, Inc in any way)

I can't go into details, but let me say that someone who would be in the know at AWS, very much off the record, told us not to use their hosted elastic because of how terrible it is.

Some issues: scaling it is manual (a "managed service" that requires direct management to scale is not a managed service lol), the elastic versions supported are usually woefully out of date, limited customer adoption means that the product doesn't get a lot of engineering attention and is just limping along, etc.

No, the IP whitelist is only to allow access to the network entry point. So if you had a VPN, it would open the port to the VPN server. In our case, it opens the port to the cordoned-off SSH-based network entry point. It's not the replacement of an existing authorization layer; its an addition where one usually isn't found.

Yes, the RDP story is very painful.

To the best of my ability, my goal was to make the post more about the network architecture (esp around the concept of SSH bastions) and less about the actual OASA product itself. I think there are a number of fungible solutions which would be just as effective (though I think the integration with Okta is a key product feature). What I find interesting and novel is more what we can do to only open ports to authenticated IP addresses, and to address connections between a single source and a single destination. To me, that's where the real power lies.

Not to be particularly combative to this top-level comment author, but I did not see a reason to reply because I did not feel they had read the post particularly closely.

Obviously defense in depth can go as deep or shallow as you see fit, given an organization's resources. We believe that the short-lived SSH certificates, IP whitelisting (via "enterprise port knocking"), endpoint authentication (device trust), password authentication, and multifactor authentication are enough to protect a single production deployment. Encompassing all of that with a VPN seemed unnecessary when other protection mechanisms like the above, and additional mechanisms that we won't speak to publicly, are taken into account.

Like with anything, it's a game of risk, and it is up to each organization to decide what risk level they will tolerate. I believe most organizations have deployed VPNs in a way that gives them a higher exposure, and simply wanted to share some of the things we have learned through the process :)

Sorry, the tens of millions of IP addresses is referencing the IP space of AWS (one of which these network access points occupies at any given point in time).

Basically. Since we are focusing on SSH in this post (and keep in mind that SSH is its own protocol, separate from TLS), it's conceptually the same: client has a certificate and a key, signed by a trusted certificate authority, and the client is also in possession of the server certificate authority. So then you have a bi-directional trust established. The certificates are short-lived, and issued after a successful authentication + dial request to the OASA service.

I have no way to prove to you that I am not some paid shill :) , but I have no relationship with Okta outside of being a customer through my employer, and I was not compensated or gifted anything for the creation of my post.

I blog about things that I encounter at work and find interesting. That happens to often be a cross-section of infrastructure and identity!

A great point. It does depend on your use case, and your dependence on manual operations. For our organization, almost all database interactions and maintenance are performed in code; if somebody is connecting manually, something pretty bad has happened. So for us, we are not really impacted by having to perform port forwarding like this on rare occasion. I completely agree that it could be much more impactful to other organizations.

I'm curious: why is utilizing port forwarding over these mutually authenticated SSH tunnels less secure than employing a VPN? From my perspective, port forwarding still adds a level of intentionality which reduces the likelihood of an incident/accident.

I wish you had come into this discussion with constructive criticism, instead of simply swinging a hammer. I, for one, am happy to learn from somebody with a number of years of experience. However showing up on a thread and spewing negativity and name calling isn't a great way to earn respect in this industry.

> The whole point of a VPN or SSH jumpbox is to airgap critical infrastructure with unknown vulnerabilities behind a hardened point of access

Yes, I completely agree. This post is literally an endorsement of that idea, with enterprise port knocking mixed in for additional security. At no point in this post do I advocate simply opening all servers to the Internet. Quite the opposite.

If you have suggestions for how I could be clearer in the post, please let me know.