The Ticketmaster Data Breach May Be Just the Beginning | Hacker News Reader

The Ticketmaster Data Breach May Be Just the Beginning

wired.com

95 pointsbelter49 comments

Snowflake is blaming its own customers for using single-factor auth and that access to the affected accounts was gained through infostealers or acquired credentials. With their recent joint statement[0] together with Mandiant and CrowdStrike, they've completely removed themselves from the discussion of having had their systems breached.

I'm not a Snowflake user myself and don't really know much about their internals, but perhaps there is someone here who can clarify if this scenario can happen:

- Credentials acquired for a Ticketmaster account at Snowflake.

- Log-in effortlessly because of single-factor auth.

- Gain immediate access to a production environment and dump the database.

Because that is how it sounds for the time being. It is Ticketmaster themselves who have confirmed that the database was stolen through Snowflake.

If the process outlined above is how the DB was dumped, then what the hell? That's the most ridiculous and crazy and easily the dumbest blunder of all time, is it not?

It's either that or there are still missing details somewhere along the way.

[0]: https://community.snowflake.com/s/question/0D5VI00000Emyl00A...

[0]: https://archive.is/acJ6p

(Archive link because the original URL can show an old cached version sometimes)

Didn’t we already read this was entirely on Snowflake? They had employee credentials leaked — Solutions Engineers specifically. These were support credentials hanging around from when large clients integrated with Snowflakes

So all it took was one master-key credential to be leaked to affect thousands of customer accounts.

The post that alleged this[0][1] was removed/withdrawn. I do think that they spoke with the real perpetrator, because there is a lot that connects with that story.

But there are a lot of smaller details that don't add up. And I don't believe Snowflake hired Mandiant/CrowdStrike just to come out with a lie that their systems weren't breached.

Maybe someone else will speak to the guy who did all this to get a much clearer overview of the situation.

[0]: https://news.ycombinator.com/item?id=40534868

[1]: https://web.archive.org/web/20240531140540/https://www.hudso...

So we’re at the “two is coincidence” stage and the next leak will change the narrative back to Snowflake.

The last exchange with the alleged hacker is a bit sus:

researcher:

> [snowflake] should have bought protection from Hudson Rock > could have saved them from this one

hacker:

> yes I agree > it would have helped

Smells like a guerilla advertisement; or a hit piece to disparage competitors. Or maybe it was because some in-house legal team pulled it back.

For now, I am getting bearish signals on SNOW. The initial reports of them blaming it on customers not enabling multi factor authentication is a way to deflect blame until they can figure out a way to deal with pending dump on their stock.

Honestly, if this exchange with the hacker is legit. The $20M the hacker extorted the company for is going to be peanuts compared to the reputation loss and stock dump.

Yet another L for SaaS based products. A centralized source of failure.

We read that[0], and then the company that published it un-published it without any accompanying notice[1].

[0] https://web.archive.org/web/20240531140540/https://www.hudso...

[1] https://news.ycombinator.com/item?id=40547921

> Gain immediate access to a production environment and dump the database

I don’t know how easy it is to dump an entire environment, but the “gain access” part depends on the user and what roles they have. If these were highly privileged users, then yes, they could get a lot.

It could be mixed responsibility here. Possibly Snowflake should require MFA (do other cloud providers do this?) but their posture seems to be “we make it possible for anyone to sign up easily”. On the other hand if Ticketmaster had an admin account with access to entire production database which was not configured for MFA, that’s definitely on them too. On the third hand, if these were regular users that Ticketmaster did not restrict access for, that is also on them.

Good questions and I've been talking / wondering similar.

I generally dislike Snowflake in this case and think they're attempting to deflect the blame off themselves for the architecture you get forced into as a tenant of theirs. I have some experience going through their docs and asking some exactly-how questions for implementations on the tenant end. I think Snowflake can both deflect blame correctly but inherently be the root cause making it hard to get perfect and right in a security context as a tenant like Ticket Master (unaffiliated to me).

If they had real dedicated tenant services available, I'd probably feel they could defend themselves here. But that'll start an interpretation of meaning argument of dedicated vs. private with folks at SF due again to their architecture and thus the only context they speak. Private or dedicated to org only is neither easy to achieve via a lengthy list of security controls nor available to buy outright ... and their ask of SF tenant's to do better about the variety of hard-to-get-layered-right controls is too much sand paper for my liking.

Another starter point on interpretation and SF's context, I've had at length discussions with them on; IP filtering such as actual packet drops / time outs at network firewalls vs. HTTP 403 errors responses at ip filtering layer, which occur at SF's shared layer7 ALB's services, actually matters. Tenant's should not have externally resolvable endpoints to worry about IP filtering or their interpretation of default deny (like their IP filtering choice of interpretation). [0]

> - Credentials acquired for a Ticketmaster account at Snowflake.

Tenant organizations like Ticketmaster here must enforce 2FA on accounts themselves proactively. Most times, regardless of 100% of interactive user accounts with 2FA, you'll still need system to system 1FA API type credentials for automation or large batch processing. These 1FA Service type accounts are usually static API key type credentials or a role where the credentials rotate but the role becomes static / assumable by another federation layer before that like in your IDP, LDAP, etc. These types of accounts still need to be usable and stored somewhere, it's a bad assumption that everyone does SOPS, key store, hashi vault, credentials manager, secrets store, and the like perfectly at all times.

I'll hold off on the variety of ways you can choose to expose and share your data to 3rd parties. I don't think it's entirely a factor but folks should worry about this in other contexts too.

> - Log-in effortlessly because of single-factor auth.

Perhaps not effortlessly find, but it's not exactly easy to actually only expose your Snowflake instance to only your organization. Variety of reasons like shared authz/n happening in Layer7 at Snowflake ensure you can't go private exposure via private-link only, without also having externally resolvable addresses existing for your instance. Thus forcing you to layer IP whitelists, again poorly chosen by Snowflake at Layer7, wherever you can find how to apply it to all your instances (prod, dev, other subdomain tenant named instances). Thus creating an open loop in Ops for maintaining these SAAS software firewall lists. And eventually always having some ALB level HTTP web servers exposed to internet doing the IP filtering HTTP:40X error somewhere at Snowflake ALB's ensuring externally resolvable at least. Snowflake's fault here for never building true dedicated org capabilities with private hosting so far as I can tell.

> - Gain immediate access to a production environment and dump the database.

Not everyone will get a 403 when finding these Snowflake doors... and when you get a 200 with an SSO prompt you can usually start with username enumeration as a feature for getting true/false exists. These tenant's are almost always subdomain resolvable, picking one at random https://epa06486.snowflakecomputing.com/ exists and happily receiving the IP filter at HTTP 403. You can find more using tools like https://dnsdumpster.com/ (unaffiliated) or other relevant DNS toolsets you can walk subdomains for a lot of this information. As mentioned, these always end up with externally resolvable points of Snowflake tenant's interest in my experience because of Snowflake architecture choices not the tenant's mistakes like how to get IP filtering 100% right and have confidence in what is/not exposed to login attempts.

[0]https://docs.snowflake.com/en/user-guide/network-policies

[0]https://docs.snowflake.com/en/user-guide/security-encryption...

[0]https://docs.snowflake.com/en/sql-reference/functions/system...

[0]https://docs.snowflake.com/en/developer-guide/sql-api/authen...

These companies are allowing customers to be defrauded. They must be held accountable and punitive damages that severely hurts them must be applied.

Which company should be held accountable? Snowflake or Ticketmaster? It's not as straightforward as expected

Customers have no direct relation with Snowflake, so they should go after Ticketmaster. Ticketmaster can then decide to reclaim their losses from Snowflake or some kind insurance they may have, but that's not their customers' problem.

I’m trying to imagine the size and complexity of the indemnity clause in Ticketmaster’s contract with Snowflake, and the number of hours and redlines and delays, .. all the lawyers. All the delays for the analytics org, all the pain, all the suffering and now … it’s game time.

This is how you shut down people from starting a business unless they are super rich. Liability first thinking is gumming up the gears of society.

Yes, those pesky regulations. Now you have to have a several exit doors if you want to open a nightclub. It keep it for the rich people.

Unnecessarily cynical reading. If the nightclub was certified to have sufficient egress by the relevant qualified authorities, then that should be enough for the business owner to not have to worry about being liable for it.

Similarly, if a food truck uses a point of sale service, the food truck owner should not also have to be knowledgeable about evaluating the point of sale’s database security. Their job is to make food.

If the government wants to protect people’s data, then it should set the standards or certifications for the vendors (and/or go after the people who misuse the data), not lay that liability on the vendor’s customers.

Sure. But this still has to be decided by a court of law - did the defendant follow all the regulations? That is what most of these cases come down to anyway - did the defendant do a reasonable job of following established norms (and regulations) to prevent harm? Are we saying that the courts should not decide this and it should be left to the government to just certify and that should be sufficient to throw the plaintiff’s cases out in all scenarios?

That seems a bit of a stretch. There is a lot of room for debate around the details but if someone harms someone else then they should be liable. Otherwise there isn't effective legal feedback to stop people from harming each other. The principle is sound.

EDIT Sorry, misread the comment. I removed the part of my response related to my confusion.

The only thing you can lose is what you've invested in snowflake.

It's not like we're turning up to the house of the CEO to burn it down.

We make the laws and can choose how liability works.

If a company knowingly keeps data about users, they should be liable. You're right that this data liability might discourage companies from hoarding lots of data about users; mission accomplished, I want companies to be discouraged from keeping lots of data about me.

It would be fine if the business of Ticketmaster just didn’t exist.

People bought tickets to things in the 90’s. It isn’t very difficult. The only thing Ticketmaster does is make that more convenient by keeping track of everybody’s payment info. If they can’t do that safely, they don’t even satisfy their reason for being, they don’t need to exist.

i bought tickets to things in the 90s. from ticketmaster. and in the 80s. the first time i listened to a pearl jam album it came with a diatribe railing against ticketmaster.

for better or worse, the industry as we know it today could not exist without ticketmaster and companies like it.

It would exist just fine. Because the leeches have been there since you and I were buying from them in the 80s doesn't make them necessary. That's just lazy thinking.

oh yes, it’s definitely outside the realm of possibility that there could have been fundamental shifts in the economy and consumer expectations that make it impossible to roll back the clock to a time when eventgoers just showed up and hoped there was seating available. how silly of me.

> The only thing Ticketmaster does is make that more convenient by keeping track of everybody’s payment info.

And charge exorbitant service and "delivery" fees.

So it's OK for companies to collect users' data, but they don't need to take any responsibility for protecting it?

SEC now requires all companies must report their breaches on their filings. Maybe that should reflect also a way to hold companies accountable via some kind of fine based on a threshold of the damages done.

Private companies do not have an SEC reporting obligation.

In the US only 1% of all companies are public.

What % if by revenue?

Or by breaches, but then of course that will be hard to gather.

Is this number even known? How would all private revenues be calculated?

Let me tell you about the large five-figure sum of money I lost shorting Equifax after their breach...

...if you expect accountability for this kind of thing it's never coming.

That single EC2 box with a couple hundred GBs of RAM running DuckDB is looking better and better each day, compared to getting billed up the butthole for snowflake...

far cheaper, yes, but duckdb leaks just as bad if you compromise credentials.

also more than just an ec2 but that is always the risk. someone describes an idea in short form, someone else with insufficient experience tries to implement.

How does that work for 100 data analyst users running reports and analyses?

Given what we've seen here 95 of those 100 analysts aren't employed in the original company.

Is there a built in/plug in feature for dashboarding and viz for duck db? Would be more appealing.

There's Metabase.

Previous HN comments indicated this could just be demo snowflake accounts, which were all compromised from a single individuals account at snowflake. But the announcements don’t seem consistent with this. Do we think propective customers really shared 100s of millions of real customer records for demo accounts? Or more likely the sales person was granted access to production systems by the prospective clients, so their credential without MFA could be used to access many customers real data? I struggle to see how snowflake can blame the customer here; secure by default is something a customer should reasonably expect for their money.

I think if it’s one customer you could maybe blame the customer and get away with it. If it’s multiple at once, all those customers very obviously are just pointers back.

It seems unlikely that a French SE would be working on Ticketmaster and Santander.

My guess is that it went down like this. Ticketmaster gave access to their production tenant to sales engineer that was probably attached to their account rep. He got an account with a set password, was not onboarded into their Okta/Azure AD/etc and didn't have MFA enabled for his account or was restricted to a range of IPs for access.

He got p0wned and the hackers got in using his creds. Of course he likely had accountadmin or something highly privileged since he was likely routinely asked to look at random things at Ticketmaster... that too didn't help.

https://archive.is/gOdBL

Not a single data breach resulted in the guilty company's demise or even noticeable financial or reputational damage. Nothing will ever change and this will keep happening more often.

I don’t get how you can just use production credentials in these demo systems. Even if they don’t get leaked it would mean Snowflake employees have access to your production data?

I’m working in the compliance space in Europe and here it would be pretty wild if a large corporate would give such kind of data access to a cloud company without tons and tons of red tape, US companies seem to be way more lenient about this.

You might be misunderstanding (or I am). My reading of the situation currently is that the “real” breaches were due to actual production credentials leaking which were used to actually access production. Then some customers were not using MFA on their Snowflake account so credentials were all that were needed.

The demo accounts were associated with a Snowflake SE which was just one part of the attack.

Cloud computing!

Store data on someone else's computers instead of own computers.

Benefits outweigh costs?

I don't care who fucked it up. But not having MFA turned on should be reason enough for everyone to be sacked. So lazy

Are they using temple OS to host their site?

When will the shaolin temple start teaching rsa and transformers ? When Kwai chat kane can come save the day ?

Reply on news.ycombinator.com