Hackers claim to have breached Okta systems
twitter.com
twitter.com
I checked their telegram group and I can see that they specifically recruit people with access to VPNs/internal support systems. This okta breach seems to have happened through similar means. The group has made specific calls for access to gaming companies, hosting providers, telcos, call centers, and bpm providers. They offer payment.
It doesn’t feel difficult to see how some overworked, underpaid support agent (or even a well paid disgruntled one) might decide to go with this. Just takes 1 well placed agent to give creds and this group has access to a huge attack surface instantly. This might sound like an overreaction, but corporations in the future might need to make least privilege access and access logging everything a priority from day 1.
https://t.me/minsaudebr/162 (Link to the recruitment message)
> If you receive a credible bribe and report it immediately to X we will report to relevant authorities and provide a bonus equal to the value of the bribe (capped at Y).
Only if you assert that it is only useful if it solves 100% of the cases.
There are other motivations, but money is a big one. If you could block X% for a large value of X (I'd say easily > 50) then you're better off than before, even if some insiders driven by other motivations still remain.
Still your point stands that money is a common factor. However you still have the problem that bribes usually aren't a one time payment but an ongoing thing, so a purely financially motivated adversary would take the total value into account.
Most people worldwide will take the moral equivalent of a bribe. It may require a sum so large as to be implausible for it to ever happen, but anybody that tells you they wouldn't at least consider illegal activities for life-altering amounts of money is statistically a liar. Millions of people have done and will continue to do it for seemingly trivial amounts of money.
The prisons of your country are full of such people.
I see only trouble with this approach.
Just hire good people, treat them right, have sensible audit and monitoring procedures.
It's very difficult.
edit: By which I mean 1) audit your processes and 2) get to know your people.
Processes require good engineering to thrive; people require good management. When you confuse the two, you get problems.
If you reported a fraudulent bribe, which was reported to the police there is a risk the briber and you both are arrested.
Some will take it as a sign they should poke further your system, or shop around for others paying better.
You wouldn't know how insulting that is before going through a good of paperwork that legally tames your desires.
That's kind of a weird moral take on things. If you don't like the bounties offered then simply don't invest time in the platform looking for things to exploit.
Industries ex software know that since time immemorial.
Prob the software way of solving that is with least privileged access, automatic auditing and approvals.
I wonder why not even security companies are doing that correctly though.
Why should a business care if it gets hacked? It's the users that suffer and they don't have an alternative.
Similar to how IT in many large companies nowadays runs internal phishing tests.
That changes the risk/reward, so while every reported bribe is a nice cash bonus, accepting any received bribe becomes significantly riskier, and by 'practicing' bribe reporting it becomes the default reaction.
This raises a number of architectural questions for Okta:
- Can a support engineer for customers in one region access the systems/databases relied upon by customers in other regions?
- Do employees get notified when their account is logged into or used (e-mail/SMS/otherwise)?
- Do customers get notified when a support engineer accesses (with privileges) a system/database the customer relies upon?
[1] https://work180.com/en-US/for-women/job/215617/tier-2-techni...
My (admittedly outdated) experience is that this is always the case, regardless of any assurance vendors might have provided to the customer. In the end, people have to get stuff done, and offshore bodies are just too cheap to pass on. Things might have changed a bit since GDPR but, in practice, I expect there will always be "channels" for people to reach out cross-region.
Just because a customer is based in Europe doesn't mean they don't need to get a hold of support at 03:00 local time. Even small, growing, startups now have HQs in one part of the world and staff across the globe due to the ease of movement many enjoy.
A single support account reading say 10x the normal number of accounts per day, with a different geographic distribution than the norm, should ring alarm bells. It’s not enough to say “well shucks they have access to the data, guess we’ll do nothing”
Otherwise, the goal is to give each agent access to an arbitrary subset of users, so it's harder to recruit an agent who can access the "target" victim customer.
* Balancing the assignment of users to appropriate agents
* Less flexibility for agents to be specialized (instead of an agent responding to all X related tickets everywhere in the world, an agent must respond to X, Y, and Z from these specific users)
* Handling user coverage when agents are busy, PTO, etc. The spirit of splitting agent access would mean agents would need to request access from an out-of-region authority in order to help with things last minute. An authority who will be expected to be responsive 24/7 without also being an attack vector
Technically, the current model is a full M customer x N agents mapping. All customers served by all Agents. Adding some 0's to that mapping should certainly be possible without sacrificing much from current system. No way to know how many is reasonable
All that said, there is a natural logic that arises from this 'analysis' - Limit access to customers who are inactive, or who rarely have issues. A common sense addition would be to raise their default priority for when they do interact.
The screenshots show that LAPSUS$ had access via the account of the outsourced Costa Rican based Tier 2 TSE worker to view details of and reset the password of a UK-based site reliability engineer of a well known Internet global infrastructure company. This level of access in itself doesn't sound hugely problematic on its own if it weren't for job applications for Okta Tier 2 TSE workers indicating server/network administration responsibilities. There is some publicly published information that provides more context on what access a Tier 2 TSE worker may have to customer data:
- Okta's security whitepaper [1] states that multi-factor authentication is needed for administrative access to AWS instances (host operating systems) so this could have possibly prevented LAPSUS$ from using their access to do anything more sinister than perhaps causing customer organisation-wide outages by disabling/resetting all their user accounts. The response for control AC-17(06) at [3] indicates that remote access requires valid multi-factor authentication to occur so if LAPSUS$ could gain remote access, perhaps they had already gained the ability to generate valid multi-factor authentication tokens, or otherwise bypass multi-factor authentication for remote access.
- The response for controls IA-02(09), AC-06(07), PS-04(01) and SA-12 at [3] indicates that Okta employee access to customer data is deemed privileged access and that fewer than 30 Okta employees have this level of privileged access to this customer data. I doubt this is true, unless the SP 800-53r4 responses at [3] are only referring to the US government tenancies that are hosted separately from tenancies of global companies.
- The response for controls SI-04(12), SI-04(19) and SI-04(20) at [3] indicates that changes made to a customer tenancy (for example resetting passwords for customer users as shown in the LAPSUS$ screenshots) should have triggered events being sent to both Okta's Splunk SIEM system and also to a customer SIEM system if they have set one up. Theoretically this should quickly raise a red flag to the customer if LAPSUS$ were to use this access to bulk deny the customer access to their ICT systems. It is however not clear that privileged access to the host operating system for the AWS instance would cause an event to be sent to the customer's SIEM system.
== Level of segregation of customer tenancies and whether LAPSUS$ may have been using Okta access to attack customers ==
It's fairly straightforward to enumerate the Okta hosting arrangements from DNS records as listed below. The sample targeted company in the screenshots has their Okta tenancy hosted within AWS region us-west-2 (Okta OK7) and other screenshots showed that the compromised account had recently accessed privileged access tools for OK3, OK8 and OK12. Thus it would appear that a Costa Rican-based Tier 2 TSE worker would have access to customer information within perhaps any of the clusters (not geographically restricted).
ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com - response when a tenancy is invalid/doesn't exist
ok2-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global)
ok3-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global)
ok4-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-1.amazonaws.com - companies (global)
ok5-crtr-tls12-fips-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - US government agencies
ok6-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - companies (global)
ok7-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - companies (global)
ok8-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.ap-southeast-2.amazonaws.com - Australian government agencies and some other Australian companies/associations
ok9-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.eu-west-1.amazonaws.com - unknown / no customers found
ok10-crtr-tls12-fips-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - US government agencies
ok11-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-east-2.amazonaws.com - unknown / no customers found
ok12-crtr-tls12-nlb-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com - unknown / no customers found
microsoft.okta.com - likely customer - ok3-crtr-tls12-nlb-dfef298ffc8f82ca.elb.us-east-1.amazonaws.com
nvidia.okta.com - likely customer - ok2-crtr-tls12-nlb-acd62f33a2a6a463.elb.us-east-1.amazonaws.com
vodafone.okta.com - confirmed customer [4] - ok4-crtr-tls12-nlb-29367a8e4bb80716.elb.us-east-1.amazonaws.com
samsung.okta.com [5] - possibly not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com
ubisoft.okta.com - possbily not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com
lg.okta.com [5] - possibly not a customer - ok-crtr-tls1-nlb-36253386bec48ce6.elb.us-east-1.amazonaws.com
[1] https://www.okta.com/resources/whitepaper/okta-security-tech...
[2] https://trust.okta.com/security/
[3] https://www.okta.com/resources/whitepaper/using-okta-to-prot...
[4] https://miniorange.com/atlassian/atlassian-single-sign-on-ss...
[5] Note: also tried similar names known to be used by company divisions.
No. It's a tricky situation too, since even if one did receive such email, I won't look at work email until working hours. So even with such a notification, if you logged in on friday 20:00, you've over 50 hours to do your work.
I might also add that Okta DOES NOT support any form of physical 2FA (e.g.: yubikeys), only software-based ones.
Not sure where you got this from, but we've been using yubikeys for a few months now. They even allow policies to only allow physical 2FA methods.
You might be operating under outdated information. I absolutely use a YubiKey with my Okta account.
As an industry I don't know if we have made rational choices along the way.
A portion of our industry seems to be bad at risk calculation, but that's not inherent to the SaaS model - this app very well could've been made with strictly separated tenants.
e.g. nobody got a promotion because the change they pushed for 5 years ago prevented an attack today.
That's not to say people are just greedy and only chase what gets them a promotion, but replace "promotion" with "received positive feedback".
This is cope - managers repeat this narrative so they can blame some other factor rather than having to put in the work to fix the problem of hiring good FTEs. Unfortunately, this narrative has been repeated so often that it is perceived to be axiomatic, which is why it is repeated without any criticism.
No, its not that hard to find good FTEs, you just haven't tried hard enough, or you're not paying enough, or your glassdoor reviews are bad.
this is not a negligible barrier. Sure you could argue that a business that isnt successful enough to hire for that expensive FTE maybe shouldnt exist at all, but a lot more is possible by being able to outsource it. And there really isnt a good reason why it cannot be outsourced from an operations standpoint - it is entirely a security problem. It's not like auth is hard to do for a third party.
> well-tested, audited and certified software package
Who is selling this?
(not a criticism of the comment above me, btw. not point I was making.)
[1] https://www.theregister.com/2022/03/21/microsoft_lapsus_brea...
[2] https://www.wired.com/story/lapsus-hacking-group-extortion-n...
[3] https://analyticsindiamag.com/lapsus-hack-leaves-nvidia-in-a...
That other product/service was compromised, which let hackers hack into MS employee’s computer, which gave them access to Azure infrastructure.
Lesson: ask your (in this case Microsoft) employees to not use work computers for any personal use.
/me goes to check if my password manager company uses Okta.
This is just wild speculation to make up a scenario where Microsoft could be affected by the Okta hack with no evidence to back it up.
I was just trying to come up with a threat vector to explain how it might be possible.
And sometimes there are chains of dependencies that need to be swapped out in sequence.
So from acquisitions alone it would seem reasonable to expect that Microsoft at any point in time has at least a handful of usages of certain SaaS offerings, even if MS themselves offer their own versions of those services.
That said, the surface area of these "unofficial" 3rd party systems is likely to be quite small in comparison to whatever their official tech stack looks like (which does -- I would assume -- centre around Azure AD). I'm not expressing an opinion on whether the GP is correct to make this link...
I've worked with some very large companies where internally they (surprisingly to me) make use of competitors' products even when that company makes their own version of that product. There are all sorts of legitimate reasons why this happens.
https://gizmodo.com/microsoft-investigating-potential-lapsus...
Honestly I’m so unimpressed with AWS that I find it hard to say Azure is terrible, because the bar for terrible is set much lower.
[1] https://twitter.com/BillDemirkapi/status/1506109956298317830...
Keycloak, for one: https://www.keycloak.org
I've also heard good things about Hydra: https://www.ory.sh/hydra/
Depending on your needs, an SSO proxy like Vouch works well for internal services: https://github.com/vouch/vouch-proxy
There's also an abundance of libraries for different programming languages to implement authentication or authorisation, lots of them battle-tested by thousands of services. It's not like providing secure services is impossible without Okta.
For medium to large enterprises, the calculus is harder and there's going to be months of flame war trying to hash that out.
The problem is motivation to break it trends up over time as the service gets more and more popular.
In some ways, if Okta can fall basically anyone can. Sometimes there's an advantage in being smaller. One would hope that the actual attack surface on Okta was smaller than the post seems to imply. That there is even any way to get full access to all customers would seem to be a critical failure.
This might be true, but for someone to "fall" they must be targeted to begin with.
If "LAPSUS$" is after big companies with significant IP which they can hold ransom for millions, as a small business you'll be safe because they won't even bother attacking you, and may not be willing to put the same amount of effort into breaching your small company's SSO than they would for Okta.
Yes, it's going to cost more upfront, but there are a wide range of options between "rely on a blackbox PaaS/SaaS" and "write and deploy everything yourself".
Maybe a company that specializes in this kind of thing? Like Okta?
How do you have an SSO system without a centralized service? Whether you buy it as an outsourced SaaS service or run it yourself, it's still SaaS, but now you're on the hook for managing it and keeping it secure.
Or for comms, run a team- or company-wide chat server rather than relying on Slack/Teams/Discord.
Either you're missing my point (and maybe I didn't express myself super clearly), or you're pitching a false dichotomy.
> who has more expertise at building and running an SSO service than a company that's selling that service
OP illustrates well the consequence of that line of reasoning. I'm certain this is just the beginning as more too-big-to-fail vendors get compromised.
The cloud is just somebody elses' computer, "Move fast and break things" mean things get broken, etc.
Whenever we get subscription services for "smart security" involving cloud-connected cameras, AI-controlled locks and drones, would you also consider that as the one-size-fits all over having security guards or contracting companies like Securitas or your local equivalent?
> now you're on the hook for managing it and keeping it secure
You always were. There is no free lunch.
If Okta gets away with this (and the Equifax breach and subsequent outcome suggests they will) it proves that the market doesn't actually care about security, thus a company selling that service has little incentive to actually invest in security.
On the other hand, I feel fine using GSuite for SSO because we have a much better view of their security.
That said, it sucks that everyone is so fucking bad at security. I maintain that it is not that hard.
Add in systems that were built for X and now doing Y. This is hard to get right. A single slip up will lead to you being compromised.
Yeah, people need to stop using memory unsafe languages. They choose not to.
> A single slip up will lead to you being compromised.
Not if you add multiple layers of security. Like sandboxing. Or mTLS. It's not hard to do that.
edit: Let me clarify. Security isn't hard generally, but it's hard individually, because you're drowning under everyone else making it artificially 10000x harder.
Golang and Rust are not magic bullets that makes systems automatically secure.
Flaws in application logic have little to do with language choice.
Also consider the effort and money it takes to rewrite a multi million LOC system with several dependent apps. The new trendy languages introduce breaking changes, switch paradigms, and have less mature ecosystems.
What are their usual criticisms? Genuinely curious. I've always hated their UI, their docs, and lack of real support but felt that at least they had security going for them. The last assumption might have been proven wrong tonight.
Also, what would be a reasonable alternative to Okta now that their nearest competitor, Auth0, was acquired by them. Is GSuite a reasonable alternative for SSO with multiple different providers and supporting SAML, etc.? Thanks in advance.
I wouldn't want to make a recommendation, but I'll say that my company uses GSuite.
I'm struggling to see how that attack works without a copy of the private key.
Any pointers for what to search for to understand this better?
A better solution is to use ephemeral SSH keys generated using an SSH CA. This kind of thing can be implemented with Hashicorp Vault, though I'm sure there are plenty of other solutions out there.
It simplifies key checking on servers too, as they just need to the details of the signer and there's no need to juggle keys in LDAP.
I don't remember Google having any large-scale security incidens.
Have they been better at playing down their security incidents or are they doing something very right that the rest of the industry can learn from?
yes
> that the rest of the industry can learn from
unlikely
It's simple: they pay absurd amounts of money for top talent and let them work. Look: https://www.levels.fyi/company/Google/salaries/Software-Engi... can your company afford to pay 2-300K cash -- not to mention serious stock -- to bread and butter mid level engineers?
Then their SNMP mib doesn't work properly. So you have a a box that is proxying some of your most critical systems that are so old that to integrate them with MFA requires OAG (think mainframes, old ERP systems etc) and you have to take Okta's word for it that the server is secure and not hacked.
They do thankfully support Syslog for logs, but again you have to take their word for it that you're getting all the logs, because you can't access the system to verify.
Having said all that. OAG solves a very real business problem and it is hard to find a competitor in the market with the level of integration it has to legacy platforms.
Security isn't just about overflows and injections, outsider enemies vs insider allies. Any human with privileged access that can be compromised will eventually be compromised as the perceived value of his privilege increases beyond the cost of compromising him. Logging, distribution of privileges, and other such solutions aren't really solutions so much as just a sort of cat and mouse game.
I would claim that impenetrable security is not only hard, but ultimately impossible.
If you are in the 5%, great! You are probably set already.
If you are not, overreacting isn't going to help anyone. There's no need to start writing user login code tonight.
I agree. That being said, there's no need to write login code yourself. Trust on the existing, tested, proven OSS solutions and libraries available, if you decide to not rely on a provider like Okta.
Can you recommend some?
I've also heard good things about Hydra: https://www.ory.sh/hydra/
Depending on your needs, an SSO proxy like Vouch works well for internal services: https://github.com/vouch/vouch-proxy
There's also an abundance of libraries for different programming languages to implement authentication or authorisation, lots of them battle-tested by thousands of services. It's not like providing secure services is impossible without Okta.
I think it's worth pointing out that replacing Okta for most people isn't just doing login +MFA stuff.
Sure, there's that part of it - but a lot of the value in going with Okta in particular is that so many third parties support integrating with them in ways that other Identity providers are just not supported.
Account lifecycle stuff (via SCIM) is often limited to just Okta support for products.
There's also the account lifecycle stuff so that HR doesn't have to tell you when they hire someone, they just add them into the HR system and all the things flow out from that. Similarly when someone leaves, you can have those workflows go and disable/delete their accounts.
In fact, a third-party’s “Okta integration” should work out of the box with Keycloak or any other IdP.
It also makes user lifecycle someone else's problem. Very important. As a medium and larger sized company, when you have to tell your customers your homegrown IAM was pwned, you're in for a world of hurt. When you're just in the boat with everyone else who got popped, it's an easily recoverable incident.
It doesn't really matter who runs your auth service, if the auth service getting pwned results in a reportable attack, you're in for a world of hurt.
The next step is to articulate why this isn't your fault and that you did everything you reasonbly could have done to not get pwned.
Do you have evidence to support you did your due dilligance on your provider?
So for example if you want to use AWS SSO tied to GSuite, you have to manually manage users, group membership, etc inside AWS SSO.
Similarly for a bunch of other products.
A couple of companies I've worked for now use GSuite as their primary accounts, and while a ton of things authenticate through GSuite (either as a built-in integration, or as SAML) - when you onboard/offboard users it involves going into a bunch of different systems to create/invite, set up group memberships, and then later deactivate accounts.
We're still considering Okta to solve all that workflow stuff. Yes, even after this event.
I believe Okta can also do custom security flows that GSuite can't.
So, if you're on a trusted device+network, then you might not need a 2FA challenge. Or conversely if you're trying to access a sensitive service - then always require a 2FA challenge every X hours.
though that sort of answers a followup Q, it would seem that there would be a market that punts the authentication to gsuite's saml, but automates the user creation and the like on the different services (and deactivation), but one still wouldn't have the "custom flows" as you said.
but if the custom flows aren't that important, I'd rather trust google to handle my authentication than okta, and in my pseudo-okta without authentication scenario, even access to my pseudo-okta would authenticate via google.
This reads to me like you are unfamiliar with keycloak, which is a very mature and capable solution with plenty of support for everything you mention other than SCIM.
Hosting it is not overly challenging, running it in a container would suffice for most companies. Updates are in my experience a smooth process.
I'm not claiming it's as easy as an "auth as a service" platform, but as we see, that comes with its own costs.
You will have to find someone to host and patch it, but we shouldn't pretend it's up there with write your own crypto.
Using saas is fine too. Often there cheaper than hiring a person until you need said person anyway, and at that point they're just a single point of failure, and people do underestimate the effort to run a production service well.
But we shouldn't be so anti-intellectual that we pretend that it's the only way to do things, and that well supported foss products don't exist.
[1] 2FA/Time based OTP: https://www.keycloak.org/docs/12.0/server_admin/keycloak-ima...
[2] LDAP/AD (Kerberos) https://dmc.datical.com/z_resources/images/assets/screenshot...
[3] SCIM (third party) https://github.com/Captain-P-Goldfish/scim-for-keycloak
Most of the time I'm talking about SCIM outbound, so provisioning/managing users in some other random service (like Trello, Github, whatever).
That's the major value-add for Okta.
https://stackoverflow.com/questions/58566587/does-keycloak-s...
So will likely require some changes to Keycloak internals in order to address that reliability issue, which will enable building a reliable interface for your use case.
Okta is a security company, and due to a well-organized, well-funded attacker, they were able to be breached. If a similar attacker is trying to breach you, you're likely going to get breached.
I think the logic here is accepting that if an attacker is trying to breach you, you’re likely going to get breached. But, if an attacker is trying to breach someone else, you’re still safe(-ish). With a centralized service, you don’t have to be the target to feel the effect.
At small company sizes, where dedicated security team or an admin with sufficient expertise to do proper setup is unaffordable, risk of Okta breach is lower than risk of automated custom setup breach. When you are big enough to deploy self-hosted solution and maintain it for reasonable cost, then yes, I would agree with you, self-hosted setup may reduce risks. When does that happen? When your company has over 1000 people (to break even with your Okta spending) or when your revenue per employee is high enough to spend every year 200-300k€ just on security.
That said, I would think Keycloak would fall under IT in many orgs. So you need a security org mature enough to have trained IT to the point where they can successfully run something this sensitive and security can have confidence in everything around it.
At a guess, that's not before you have 15-20 people in the security org. Probably something like: alerting and incident response (i.e., ops) is 3-5, vendor management 2-3, governance/audit 2-3, security engineering 3-5, security architecture 2-3, management 2-3, privacy 1-2. Run something like this for a year or two and see how the business as a whole shapes up.
As you say, it's a cost question. At what point does running your own IdP become the most valuable thing your company could do with a given chunk of money? One of the perks of a vendor is that the contract comes with liability. How much is that worth to the company?
Your company isn't delegating any operations away? Not even support? If they are, what level of access do those people have? Are the internal services they have access to as secure as your perimeter?
Even if they aren't, the attack can come through any internal employee, and your least security conscious person is your weakest link. Again, what do they have access to? Will they be able to use them to move laterally? Spear phishing from an internal employee to another employee is really, really effective.
BTW Keycloak.current has tossed JBoss & friends for Quarkus. In any case, it's been a remarkably safe product over the years.
Proxyshell and similar recent issues have shown “near-instant compromise” to be the current state of affairs. Most instances are attacked within hours or even days before a vulnerability is disclosed publicly and hits the news.
If "LAPSUS$" is after big companies with significant IP they can hold ransom for millions of dollars, it's most likely not even worth their time to find let alone attack your small company's on-prem SSO even if it was a single shell command away, so you'd still be more secure.
Wait, what exactly makes this problem so fundamentally hard?
What stops someone from just releasing a software system that solves the authentication infrastructure problem in a general way (e.g. as Kubernetes did for cloud orchestration)?
If you mean that a single codebase could be an attractive target for attacks, it's still not a problem since the Free Software community loves having a dozen of competing implementations of the same thing just for the sake of it :)
And this isn't even solved by having standard protocols, open source implementations as you mentioned -- only mitigated. OpenSSL was exploited too.
Kubernetes doesn’t go after it, either; it complicates your system so exponentially that it drives the same lie. All Kubernetes does is assign a workload to a machine and hook crap up to it. That problem was solved in the 1960s by systems that don’t take multiple FTEs just to operate itself. When you look at Kubernetes in this light there’s a moment of clarity waiting regarding why it exists in the first place, and why it will never in a millennium replace Borg.
In 2010, I joined a social media startup. I was the only operations employee on the team of nine who had ever touched an actual server with my hands. We’re on the third or fourth generation of systems people now who’ve never heard anything except the cloud-native drum and how they’ll never be qualified to run their fingers along the faceplate of the computers running their business. paxys is sharing a derivative of that because it is a strongly held industry belief and a safe language to speak in this market. It just doesn’t make it true, and that’s a powerful realization, but that’s not something most people are ready to hear and truly understand.
You’re asking the absolutely right questions. You’re just asking an industry that gave up on itself so long ago that kids born after UTAH2000 can vote now.
[1] https://www.thediff.co/p/plaid-data-layer-to-payments-layer
In a B2B setting you get rid of a LOT of paperwork and due diligence by just saying "We use Okta for X, Y, Z." Hell for SOC2, Okta got rid of a shit ton of verifications for us at a critical stage of the startup I was leading, Papa.
Sure you can build your own but at that point you're wasting months for zero customer benefit. You could be building product. I don't know what the right answer is.
No, not just that.
Commercial open source companies (COSS) minimise those tradeoffs and give you the best of both worlds. That is - The power of build and the speed of buy.
You can migrate to and from a completely self hosted solution to a managed service as your needs change (for many COSS alternatives)
Agreed. Centralization of attack surface is a major risk.
I understand the allure and have been in situations where even knowing the risks, the path is forced upon us because "nobody is fired for buying IBM".
But yes, a centralized service holding all the marbles is an infinitely more attractive target than lots of disjoint little systems which aren't individually very interesting. The centralized system may be better run (arguable, but at least they have the resources and it is their primary job) but they also face a much higher attack concentration.
A compromise solution is to have both. I've been in one company where okta was used as the primary SSO and for most internal systems the only one. But for the very sensitive resources, one had to be authenticated via okta but also via a homegrown MFA system.
It doesn't work.
In my book that counts as "obscurity". "We are not an attractive target, hence we won't be targeted". You will be targeted, as long as your defence is weak. Only if you manage to set up a comparable defence to a large provider, this argument flies: because all things equal, attackers will attack the more attractive party. But all things are not equal. (Which I'm not certain about, it may very well be that Okta has a really poor defence in place, in which case my argument falls apart, because you'd be able to "make all things equal" much easier. I doubt this though).
> Are we really at a point where not relying on external account providers is considered unusual, or a bad practice?
No. I consider it reasonable to "build your own", but only in certain cases. Things like auth are not your core domain (unless you are an auth-provider, obviously), so all effort and time you spend on building the umpteenth login/auth-flow is not spent on building the stuff that sets you apart. Even if you drag in a standard library and only spend an afternoon: you're still maintaining it, testing it, etc. And with external parties, generally, you'll be following market best practices: fingerprint-login becomes standard? You'll get it almost for free, whereas in the diy-case you'll be designing, testing, building such progressing tech for ever. The economics for building are just wrong.
As are the security: as pointed out above: I highly doubt you'll be able to match a level of defence a large, focused, experienced party can obtain.
But there may very well be cases in which building your own login/auth makes sense and the tradeoffs in economics and security (and features) make sense. Maybe it has to integrate into some legacy; maybe (legal) requirements enforce you to keep it all in-house, etc.
I find this point to require more subtlety; "we are not as much of an attractive target, so we likely won't be targeted by a sophisticated, human adversary" is more apt for the situation. The attack vector is just not comparable to Okta's scale, which is true for the majority of companies and platforms out there. That doesn't mean security should be taken lightly, just that the magnitude and frequency of bad things happening will be lower.
> Things like auth are not your core domain
That's a good point, but it depends on the situation. I wouldn't recommend spending lots of time on this yourself, but there are standard libraries or OSS solutions available. If you're optimising for reliability, it might be beneficial to keep auth within your own systems. If you're optimising for privacy under GDPR, for example, it might be beneficial to not trust your users' data to an external provider. And so on.
Unfortunately, this is in direct opposition to
> That doesn't mean security should be taken lightly
I agree about the subleties that, indeed, make the tradeoff more nuanced. You are less likely to be targeted. But, given that you "wouldn't recommend spending lots of time on this yourself" (my recommandation too), the trade-off balances dangerously into the "attractive target" too.
> but there are standard libraries or OSS solutions available.
Which, unfortunately is hardly different from "a centralized target". Look at, for example, WordPress. I used to have a WordPress hosting company: the constant battering on the entry-doors, by bots, and humans is real. The amount of successful breaches are staggering. I don't host anything WordPress anymore, yet IPs that haven't ever seen WP are hammered by bots trying to breach WordPress admin, XMLRPC, Woocommerce and so on.
By your reasoning, none of the WP sites we hosted back then were "attractive". By that reasoning, the servers I run now, without WP, are even less attractive. Yet they are battered by attackers. 24/7. And breached (not my current servers, though) quite often. Small websites for coaches, mom-pop-shops, the garage around the corner, your local football fanclub, and so on. Breached. Because they were an attractive target, not by being centralized, but because they weren't. Exactly because they use (OSS) libraries, combined with very little resources for defence, they are attractive.
> In late January 2022, Okta detected an attempt to compromise the account of a third party customer support engineer working for one of our subprocessors. The matter was investigated and contained by the subprocessor. (1 of 2)
> We believe the screenshots shared online are connected to this January event. Based on our investigation to date, there is no evidence of ongoing malicious activity beyond the activity detected in January. (2 of 2)
Up until that statement I was willing to believe that they only just found out about it.
One can only imagine the many security breaches that are never revealed (from any company).
However, if LAPSUS have more, this could easily lead to them wanting to prove him wrong and release more proof of compromise. If that happens his credibility will be toast.
Speculation from here on.
In my personal opinion,Cloudflare's actions indicate that Cloudflare was not notified of the breach until today.
```We are aware that @Okta may have been compromised. There is no evidence that Cloudflare has been compromised. Okta is merely an identity provider for Cloudflare. Thankfully, we have multiple layers of security beyond Okta, and would never consider them to be a standalone option.``` - @eastdakota - https://twitter.com/eastdakota/status/1506143353544478724
Thus, it should take approximately zero actual time to conclude what was stated here.
[1] https://blog.cloudflare.com/cloudflare-investigation-of-the-...
Edit: to be clear, I don't know, it's a genuine question. Does "any breach" count?
Otherwise, business as usual.
It's clear some service team at Microsoft used Okta to SSO, or a contractor that did that's why they only got 37GB of code and Bing/Cortona not Windows OS or internal tools.
The group probably enumerated as much access from Okta as possible and when they had every juicy target decided to release it all for the lols. They probably have more in the works too.
I'd suggest everyone today revoke any Okta SSO sessions for your apps and force a new session as a precautionary measure.
Did i miss some news or? I know Okta and Microsoft was compromised, but is there anything which shows it was related except for the hacker group and timing?
Holy red flags Batman. How can you be a serious security company while outsourcing such critical components?
1. when marketing to customers - "our support engineer"
2. when reporting on an incident - "a third party customer support engineer working for one of our subprocessors"
https://twitter.com/BillDemirkapi/status/1506117152461496323
It is a complex issue with entry of nuance I'm too tired to thumb type. I know there are lines that shouldn't be crossed. But it is so naive to think companies shouldn't have an eye on their data.
Do you think banks shouldn't have cameras in their lobbies? Or that tellers can walk in and out of the vault with black bags and no one ask what's inside?
To use your camera analogy, you better have a good lock on the security room holding the tapes.
Organizational endpoint security seems to be much more about incident response than incident prevention. That is, it's more focused on providing a nice audit trail that can be used to find and fire/arrest the perp once a breach occurs than it is at preventing breaches from happening. It probably fills a due-diligence tickbox somewhere, allowing the company to say "if a breach occurs (the probability of which is always nonzero) then we have measures in place to find out how, where, and by whom and mitigate the damage" to shareholders and important customers.
Software diversity is good. Remotely controlled mono-cultures are bad. IT management and security compliance people need to understand this.
I liken the mindset of (only we will use the remote control software to do good things) to Encryption back doors (only law enforcement will use them to catch criminals). Computer scientists call these 'Exceptional Access Systems' and it has been shown (many times) that it is impossible to ensure they will not be abused and used against you.
Keys Under Doormats: Mandating insecurity by requiring government access to all data and communications:
What alternatives are there? I honestly don't know
If you're working from home and your device gets owned, and the attacker just sits there slowly slurping data as you go about your business... Chances are you're not going to detect it, but if you did you'd probably notice it's running when it's not supposed to and it's sending network traffic when not actively used.
If someone uses a laptop breach to go look at data you wouldn't normally look at ( or tons of records ) an audit log would give that away.
So really to do security well you have to profile the role. Why are you looking up customer record abc123? Trying to see what your girlfriends doing, or is it because they've called the customer support line?
You basically have to define what does misuse look like and protect for that.
Back on the slow slurp, hardend immutable devices that are regularly updated with limited Auth sessions would likely contain the breach.
Same deal on servers, you harder to stop it, but you monitor for when it happens.
Do people really monitor though? Some, sure, but your average org buys a tool the fails configure it with any sort of context.
Software diversity also means attack surface multiplication, so it's a delicate tradeoff.
I have talked to security engineers that have had the company procure millions of dollars worth of security tools, and yet they don't even know the basics of security.
For God's sake, I had to explain what end-to-end-encryption was to a team of "senior" security engineers the other day. They genuinely had no concept of it. As a security SWE I am so done with these sort of people. Fuck them! Why are they even in security!? Real private data is at risk because of their incompetency!
It would be interesting to know Tanium takes to backport security fixes. I mean py2.7 was sunset 2020-01-01 [1].
[0] https://www.tanium.com/ [1] https://www.python.org/doc/sunset-python-2/
When I was in college the whole student network went down. Apparently there was a hack from the student network to the campus police network, so campus IT pulled the plug on any of us having network access. Because those two weren't separated because why would they be?
So I hoofed it to the IT department and I asked to talk to someone who knows a thing or two about networking. The director of IT came out. He sat me down and started telling me about the $300,000 budget he just got approval on. I was thinking, you can't take some of that and stand up a router with packet filtering to isolate the student network from the police network in two different subnets? A FreeBSD box might do in a pinch. (It was the early 2000s and network traffic loads were... different.)
I learned a lesson that day about seniority and the Peter Principle. Climb high enough and your mindset changes, from technical solutions to issues of money and resource allocation. From there, "let's just pay $VENDOR to do it, they'll solve the problem for us" is but a small step.
Same company had a head of security who was full-time employed by two different companies at the same time (he had unlimited vacation time for both) without either knowing the other existed.
Best thing was that the company did investment/insurance services and had a banking division.
0: https://twitter.com/BillDemirkapi/status/1506123471352438784
“Authentication services company Okta Inc said on Tuesday it is investigating a report of a digital breach after hackers posted screenshots of what they claimed were its internal company environment.”
It’s perfectly okay to not say anything prematurely that can cause any confusion; with the employees, customers, and media.
All eyes are on them; it’s better not to screw up whatever little trust they have left.
I really don't see the problem with their response. What would you propose in the circumstances?
It's distinguishable from a noop because some information is imparted, namely that a) they are aware of the issue and b) that they have formed a preliminary view it warrants a response.
(I'm aware he's often on here as well, hi eastdakota!)
It’s also very disappointing this is surfacing via tweets and not a breach notice from Okta, who apparently suspended at least one account used for entry.
If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic.
You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage (https://www.youtube.com/watch?v=30jNsCVLpAE). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.
>"Dead tired people are more likely to make brain dead decisions"
Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep:
https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
Like I mentioned, it's sort of an "Armageddon", or worst case scenario. It's meant to help us identify issues with our docs, assist with order of operations, and it exposes any new hires to the full scope of all the moving pieces.
We already perform DR exercises, including testing our online backup restores, between production and DR sites, but this tabletop takes it further.
Hope that answers your question.
The lawyers might be sweating on their next round of bonuses, due to dealing with the fallout. But their jobs are not likely to be on the line.
When they always have a safe haven to run to and lay low in a comfortable way without significant losses (that aren't just cost of business), what good is such a slap?
In my mind I see it like the same as throwing a beer bottle at a robbers car as they drive away. Even if you hit it, at best it's some scratches or maybe a busted tail light, but they're still gone with your goods.
Doubt it
Government is big business for AWS.
No secret or top secret info is permitted in the typical Impact Level 4 and Impact Level 5 FedRAMP systems. Okta is only certified at IL4 I think.
AWS & Azure do offer IL6, meaning information processed up to the SECRET classification.
> Can anyone who’s gotten a satisfactory answer to #Log4J #Log4Shell from @Okta raise their hand? We certainly haven’t. #rottenfishstinks
As for how often I open twitter links, is as often as they make it to HN front page. So something along of one twitter link around every 2 days.
The news of the coming days may well prove me wrong, but i am not assuming the worst from this yet. Many companies whether or not they use an idaas do things like login anomalie detecting, and users coming in from weird locations and weird times of day would be sure to set of alarm bells at some of the big targets. Heck, AWS does it for customers with guard duty.
It seems strange that such a user would have wide access. It could be that his account was just used to gain further access, or it could be that his account had wide access by mistake. Or the user doesn't actually have that wide access.
There are talks about superuser access. But is that referring to the user's actual privileges or the fact that he has access to the tool called "superuser" shown in the screenshots?
I need more patience.
Because, technically speaking, Okta advertises with their SOC II / ISO27001 / ISO27017 / ISO27018 compliance certificates [1] and [2]; which means there has to be an incident response system that would've caught this. Well, either that, or the ones that manage the incident response system are the mischiefs themselves.
[1] https://trust.okta.com/compliance/ and https://archive.ph/654MX
[2] https://www.schellman.com/certificate-directory -> search for Certificate No. "1106074-8"
Like actually being told that as reason to use Microsoft AD. Or the nobody is better dealing with spam than Google (when I was at a G Suite company). Or never to roll your own security library.
All these were in the same ballpark and in some occasions the latter one was used to drive home the others.
I am not sure what to do/say about it.
I am not (by trade) an IT person but hopefully learned a bit in my time. Trying to understand how all the different tools on my corporate device work (together?) to enhance security is impossible to me. I can't name them but from anti virus, to compliance checking to additional compliance checking by VPN software to endpoint monitoring to remote install capabilities to windows management to remote desktop capabilities and many more.
To me this sounds like too many potential attack vectors waiting to be exploited.
Any URL *.okta.com resolves and loads an Okta login screen but doesn't mean it's an actual customer.
For example, https://fake-ycombinator.okta.com works and shows the same login screen as https://pets.okta.com/. But only the latter is on the list, how do you know it's a legitimate customer?
Can someone help me out then here? I checked the domain here: https://phonebook.cz/, but manually inspecting the certificate, I don't see the * in front of okta.com to denote a wildcard domain is in use(*.okta.com). What am I missing?
Does Gartner have any credibnility other than being a pay to be endorsed shill?
So, while I realize this is a small data point, it is one backed by quite a bit of data/analysis. Their report was well reasoned and accurate given the stated priors.
I would say that their credibility is fine so long as you read the reports fully and not simply glance at their magic quadrant figures.
seems like it'd be possible write an app that scans email history to detect services that have your data, scrape their websites to determine support email address(es), and send a GDPR data deletion request template to each of the selected ones.
> In late January 2022, Okta detected an attempt to compromise the account of a third party customer support engineer working for one of our subprocessors. The matter was investigated and contained by the subprocessor. (1 of 2)
> We believe the screenshots shared online are connected to this January event. Based on our investigation to date, there is no evidence of ongoing malicious activity beyond the activity detected in January. (2 of 2)
Regardless, their statement was a legal word-soup that bounced around the issue. The Lapsus$ team responded to it and said they could reset the passwords and MFA of over 95% of Okta customers which would mean they could get into any account, and thus any service, they wanted, though obviously they would be noticed pretty soon after.
This is both sides claiming things without any evidence, mind you, so take it with as many grains of salt as you need.
I had never heard of it.
It's an auth/ID service provider for many of the things I have heard about, I guess. For internal IT systems? Or for customer (public-facing) auth?
They have plenty of business customers using it for auth to their external public customers. You can see MGM resorts, Major League Baseball, Albertson's Groceries, etc, here:
I bet a lot of IntoSec departments are stressing right now because of this tweet.
WASHINGTON (Reuters) - Authentication services company Okta Inc said on Tuesday it is investigating a report of a digital breach after hackers posted screenshots of what they claimed were its internal company environment.
A hack at Okta could have potential major consequences because thousands of other companies rely on the San Francisco-based firm to manage access to their own networks and applications.
The screenshots were posted by a group of ransom-seeking hackers known as LAPSUS$ on their Telegram channel late on Monday.
> Authentication services company Okta Inc said on Tuesday
Where? What did they say? To who? The person writing the article?
WASHINGTON (Reuters) -Authentication services provider Okta Inc is investigating a report of a digital breach, the company said on Tuesday, after hackers posted screenshots showing what they claimed was its internal company environment.
A hack at Okta could have major consequences because thousands of other companies rely on the San Francisco-based firm to manage access to their own networks and applications.
The company was aware of the reports and was investigating, Okta official Chris Hollis said in a brief statement.
"We will provide updates as more information becomes available," he added.
The screenshots were posted by a group of ransom-seeking hackers known as LAPSUS$ on their Telegram channel late on Monday. In an accompanying message, the group said its focus was "ONLY on Okta customers."
Security experts told Reuters the screenshots appeared to be authentic.
"I definitely do believe it is credible," said independent security researcher Bill Demirkapi, citing pictures of what appeared to be Okta's internal tickets and its in-house chat on the Slack messaging app.
Dan Tentler, the founder of cybersecurity consultancy Phobos Group, said he too believed the breach was real and urged Okta customers to be "very vigilant right now."
In an email, Tentler added, "There are timestamps and dates visible in the screenshots indicating January 21st of this year, which suggests they may have had access for two months."
seems pretty easy to hack a SaaS anyway. just need to phish an engineering employee account. often they have superuser access to debug customer issues
> yes we know the URL has a email address. the account is suspended - we dont care
Not saying this looks good and really not trying to defend Okta, but a user account being terminated isn't proof that Okta knew about it.
If some people are using it, I would enjoy their feedback.
There is a lot of conversation here about building your own vs using something 'centralized'. All software comes with the “build vs buy” tradeoff.
Our goal at SuperTokens is to minimise those tradeoffs and give you the best of both worlds. That is - The power of build and the speed of buy.
You can migrate to and from a completely self hosted solution to a managed service as your needs change
My reasons for avoiding any online, managed by random company, type of service like this is for the risk of a single point of failure demonstrated in this attack.
262 points by jstanley 13 hours ago | flag | hide | 162 comments
11. Hackers claim to have breached Okta systems (twitter.com/_mg_)
824 points by obi1kenobi 9 hours ago | flag | hide | 280 comments
There is no way to justify this. This smells of yet another mass flagging by interested parties and/or their network of friends. Or something worse.
Typically with the aim of shifting power from one group to another.
The world isn't getting worse, we just hear a lot more about what's going on.
People are as selfish and power/money hungry as they've always been. Humans haven't changed, there's just a lot more of us each day.
We're a long way from global peace and economic and environmental harmony but I feel like there are a lot of people who want to try.
More people than ever before and with the Internet providing means of rapid, global communication and coordination those voices are reaching further.
If we can replace capitalism with something sustainable that brings peace and equality to all, perhaps then the world will start to be a better place (but will we ever rid ourselves of the selfishness that means some people want to be more equal than others).
Why are people Nittering something they can view from source. This is why propaganda is so easy, it's what people want.