PingCap TiDB Leaks
fulcrumsec.vg
fulcrumsec.vg
I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
That does not appear to be true.
im not an expert but if I was an investor I'd try to find a buyer for my shares and run
is load bearing I expect.
> production systems
Wonder if they just reclassified all operational systems as pre-production, post-production, quasi-production etc.
No commands run.
No persistent access.
Lawyers are amazing.
Edit: Yes, based on the comment by dherls, this is how they're getting around it. Streaming replication is not commands run by the attackers (it's your own system doing the exfiltration!)
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.
[1]https://www.pingcap.com/press-release/tidb-ncc-group-securit...
There are some teams that are better than this, but they are rare.
It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.
"good opsec will save you from bad crypto. but the converse is not true, good crypto will not save you from bad opsec"
15 years ago I helped do a PCI/SOX compliance evaluation for a company looking to go public.
They had PWC in there with kids fresh out of college, no computer experience, and they would sit at the desk with the admins and say, “show me your Active Directory password policy”. Ok, looks good. CHECK
Why don’t you have this configured? I’d hear an alarming bullshit reason. Ok, we can exempt that. CHECK
I could go on and on and on with how bad it was.
And they bragged to potential investors. We are PCI/SOX complaint.
Hopefully this hack is a warning for anyone that might use NCC's services in the future.
Holy hell, I've heard of this happening, but I've never personally seen it in my 10 years as an SRE. That's absolutely wild.
With support from outside counsel and third-party forensic specialists, there is no evidence that data stored in TiDB was impacted. We keep on investigating and will provide updates on any new findings.
But what matters is that it was all memory safe and had no data races.
and how are people not talking about this more
...and now this...
> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.
This is an entire leak of ALL PingCap TiDB's customers AND data.
All of this in a single day. It is over.
Size is not a good indicator for content quality
Defense as we see it today seems fundamentally broken. The answer is to fight it with more AI? That doesn't seem right. Can software reliably self evolve and keep patching itself? What certification is going to protect you from an intruder with their legion of minions, none of them, you don't have perfect visibility to your systems, your systems emit too many signals to ingest 100% of actions, so does software need to get fundamentally simpler?
Today has me wondering how many datacenters exist out there just for organized crime? Are the big kingpins of the world going all in AI as well?
Lots of questions popping into my head today, I'm trying to think how you would slow or stop this and I can't help but think we just need everyone to go outside and touch some grass and remember we're all stuck on the same rock.
Kind of like how permissions model of deno. If you make everything closed by default and selectively open it is very easy.
Not putting credentials to everything in jira isn’t related to using of ai for hacking imo
That's okay if components are small, have well-defined boundaries / dependencies, are easy to inspect, and all that is rigidly enforced.
Obviously that's not where we are.
On the bright side: in most cases that complexity is self-inflicted, unnecessary, and can be side-stepped if so desired. Problems we're seeing don't occur by necessity: they occur because we've complicated many jobs more than those jobs need to be complicated.
Go back to first principles, cut bs wrappers around everything, and move on.
This goes to show the benefit of maintaining separations between the control plane and the data plane.
This site does say it has SQL logs which have some amount of customer data and there is a mention of some snapshots/imports- this looks like a case where the control plane and data plane end up getting entangled. Its possible the attacker could have reset customer database passwords to try to connect to them or could have exfiltrated customer data by initiating snapshots but just didn't want to deal with the increased risk of detection.
TiDB is for large scale data. You generally wouldn't use it unless you have at least 1TB of data. A 10TB leak of information is less than the size of a single small customer database for them.
> The billing engine that trusts unverified X-ACCOUNT-USER-ID headers. The account service where admin is a tenant ID and impersonation bypasses org checks with owner role.
What is the issue with the billing engine trusting an “unverified” header here? Is the assumption that it’s not auth-ing the request? Is the billing engine internal? What does “unverified” even mean in this context.
What’s the problem with Admin being a tenant ID? How would impersonation even cause bypassing an org check?
Depending on how the billing engine is deployed, and whether there are adequate controls that it's being deployed correctly, there is nothing wrong with accepting a header like this from an upstream service that produces it or verifies it in the correct manner. It only becomes an issue if the service is directly exposed to potential attackers that could use it for impersonation, e.g. a service directly exposed to the internet.