Hacker News Reader

Comment by gandreani | Hacker News Reader

A lot of people are focusing on how irresponsible the engineer was about using the work computer. That's obviously an issue but the bigger failure mode to me is the Engineer C's container deployment token.

That's a huge red-flag. In a company of this size, the deployments should be via CI only. CI isn't just to increase the speed and reliability of deployments but also to provide a clean room to build the software! Even if it's a token that was copied and pasted temporarily, the systems in place should not have to require this type of 1. copy 2. paste 3. remember-to-delete the token from my PC.

Also, Binance should be paying for intrusion testing. Better to pay a trusted company to do this kind of thing instead of letting North Korea show you how it's done. I don't think this exact exploit path would have been found but I bet the intrusion testing would have figured out another way to use Engineer C's token.

View on HN

Nobody in the entire company should have access to move $billions in crypto, much less a random engineer, even if they're part of the deploy process.

They had controls in place to prevent large transfers, so the attackers structured it as a lot of small transfers very quickly

Well, then they need another control that prevents “small transfers very quickly” happening, I guess.

Aren’t “small transfers very quickly” kind of the whole point of what Binance does?

Great hindsight! Anything you want to suggest in advance of it happening?

Not just that! The EIP-7702 delegation to make the wallet into a smart contract bypassed all of the controls that they had in place on the signer. It's wild that they didn't have a system in place to stop it from signing that transaction type.

From my rough understanding, this seems like the main exploit. It seems like the machine with access to the wallet key was presented with a crazy EIP-7702 transaction and signed it (which effectively owned its wallet in a single transaction)[1]. Without knowing their security model, it seems like biggest trust boundary should be between that signing machine and everything else.

[1] The fact that the needed to sign 50 of these transactions to solve a race condition seems like an implementation detail

They had to sign 50 of them because they had to compete timing the transaction with the normal transactions. Transactions have to be in nonce order for the wallet, and if two transactions with the same nonce are submitted to a block, it is up to the miner / validator to determine which one to put in. So if you spray 50, you can almost guarantee that the transaction you want will get in.

Is quorum signing not a thing in crypto?

In Ethereum it would be implemented by linking your account to smart contract code, then throwing away your private key so you can only make transactions through the smart contract. The approval of linking to smart contract code is exactly what they signed here - only it wasn't a multisig smart contract, it was something malicious.

It is---though I don't see the connection

A disgruntled employee wouldn't do the exact same thing?

That's the plot of Office Space!

Also Superman III, which predates it and presumably was well known to the writers. But I have an inkling the scheme and it's infamy go back much further. (Precious metal coin clipping/debasement?) It's one of those ethical questions, like a pauper stealing bread, or thought experiments, like "what if reality is just a dream" that many if not most people encounter during their formative years. Per usual, though, the chasm between theory and application (even just identifying real world potential when it matters) is the toughest one to cross.

They also mention that in the movie.

https://en.wikipedia.org/wiki/Salami_slicing_tactics

there's even a Peter Ustinov film from the 60s

Where’s Milton when you need him??

If they did and didn't account for a simple scenario like this happening and built in roadblocks and friction to prevent it from happening? Then it would stand to reason their entire EIS team should be fired.

The fact nobody thought of this is pretty staggering.

It is hard to imagine they lost 6.3$bn and are not firing people...

They just spent 6.3$bn to train them....

> They had controls in place to prevent large transfers, so the attackers structured it as a lot of small transfers very quickly

lol that's the worst control I've ever heard.

OK, so nobody else has seen Superman III?

It is such a weird combination of "I enjoyed this movie" and "this really isn't a good superman movie"

at that time I was not old enough to see it

Nope.

Be your own IRM 4.26.13!

I think they did that in Superman 3

This isn't a mundane detail, Michael!

best part of this thread so far

most employees shouldn't be able to touch the funds at all except a handful and probably require supervisor verification (dual factor authentication).

I mean sure, but as long as you can deploy directly any permissions models are meaningless for an attacker? They can just be removed.

> Even if it's a token that was copied and pasted temporarily, the systems in place should not have to require this type of 1. copy 2. paste 3. remember-to-delete the token from my PC.

One of the first things I did in my last job was shift processes like this off my team's computers, aside from the security ramifications all it takes is one dumb mistake, one up-arrow too many, one didn't-read-the-whole-line, and now you're doing something to production by accident!

I also did that, for a team deploying smart contract on chain from local.

It took a couple months to get the pipeline out of their laptop.

Worth it, yes, as we found interesting things in there which explained why they were so reluctant to deploy upgrades.

> as we found interesting things in there which explained why they were so reluctant to deploy upgrades

.. backdoors which enabled the dev to deploy exploits of their own which wouldn't be visible through CI?

No, rather a particular environment with monkey patched libraries, alias for various versions of compliers, and dozens of env variables that happen to work or not work well with the desired config.

What sorts of processes related to secret storage were you able to shift off of team computers? From my perspective, if I need to get e.g. an API key from Twitter and put it into my prod environment, i have to be doing that with my computer somehow. There's still millions of different types of secrets that need meat puppets to shuffle them around.

> There's still millions of different types of secrets that need meat puppets to shuffle them around.

Sure, but those meat puppets shouldn't be developers, they should be administrators, using proper procedures.

Separation of powers seems so fundamental yet also so totally ignored these days.

TBH CI does not prevent this kind of mistakes. I've seen people triggering the wrong CI pipeline.

I guess people triggering the pipeline is already part of the problem

No matter what you do, at some point you have some human performing some action which will, after N indirection layers, result in a change in prod, unless you have the mythical fully automated agentic software factory.

The article says that a software engineer interviewed with the North Koreans and the NKs then gained access to engineer A's machine (which had the deployment token) and then finally passed that token to run on engineer C's machine (who apparently did deployments). They knew of the team makeup from their first (caught) attempt.

That unlettered (npi) engineer who downloaded and ran the interview software on his or her machine is what the human weak link here. Let's call them Engineer F. Seems like this person was specifically selected from the dev team.

I've had this happen a few times getting interview requests from Indian recruiting firms. It sounded familiar. You go through a screening, then get passed onto the "code challenge" part. You have to download a GitHub file and then work on it, check it back in and push it so they can examine what you did.

Twice I scanned the file after downloading it to a sandbox and twice it found malware that would've been installed.

While this is by far the worst case of this happening, I've read at least a half dozen other accounts of recruiting companies attempting to install malware through the same process. I'm not saying the person should have known, but it's scary this is becoming more and more common.

> I'm not saying the person should have known

This person definitely should have known. You don't download and run arbitrary binaries outside of a VM. At this point you don't even compile code from an untrusted source. This is the power user equivalent of falling for a phishing email.

One of my favorite stories from a security talk was the speaker recounting the surrealism of (while doing a forensic investigation of a security incident) watching the attacker read your internal documentation to try to figure out how the hell anything works mid-attack, and running into all the same pitfalls as your new employees.

That suggests a creative new justification for not documenting your code :p

No such thing as deleting secrets. You decommission them and replace them. Deleting them off disk is impractical to do, at least sufficiently.

I don't even leave deploy tokens on my laptop for personal side projects. It isn't difficult to setup CI deploys, if nothing else the extra upfront cost let's me be lazy and have a consistent deploy pipeline.

100% this. I can deploy to our sandbox environment directly, but only our CI/CD has the permissions to build/deploy to our master environment. And the stuff I work on is 1000x less impactful than $billions in crypto!

Maybe that second engineer who's laptop they were laser focused on was the CI/CD?

> Better to pay a trusted company to do this kind of thing instead of letting North Korea show you how it's done.

but enterprises usually prefer the latter, I guess it's more convenient since it bypasses the bureaucratic processes?

I have saw companies doing social engineering as part of their intrution testing, I imagine fake interviews are (or will become) part of these.

I think it will remain uncommon. I agree it would be a good test, but companies and employees alike get very angry when pentesting gets too close to sensitive subjects.

Anyone who fell for it would not only get in trouble for poor security practice, they would reveal that they were job searching. That opens an entire can of worms that creates problems beyond the pentest. This also angers other employees who see it as a double-trap being sprung on employees.

Another major problem is that responsible employees who accept the fake interview but then use their personal laptop to do the interview will have their personal laptop compromised by the pentesters. Responsible testers will immediately abort, but some will try to use the personal laptop to pivot deeper which is a huge ick. It’s also really bad if the company is compromising personal laptops of people who are doing the right thing (switching to personal gear for interviews).

The pentesters wouldn't have to actually pwn a laptop, just getting the employee to download and run anything could go in a report without having the file actually be malicious.

Would they have legal right to do that, out of curiosity?

Hired oentest has company permission to do stuff on company property. I do not imagine they have right to do so to personal property.

Absolutely not. There is literally no framework, which would give you the right to exploit an employee’s personal laptop, that’s just cyber crime.

Then that is something the pentesters cannot do, but malicious actors will do.

Game over.

What pentesters can do, should do, and often do is simply start the test from a workstation that is given to them - you assume a successful social engineering attack as the starting point, and test whether your process and technical controls can properly limit the impact of such a compromise without trying to replicate the actual social engineering step and the associated disruption to the targeted employees.

Taking over that first engineer's workstation shouldn't ever have resulted in $6.3B of damages. I would consider it reasonable if those privileges could be abused (by e.g. the same engineer intentionally) to cause $6.3M of damages and the rest would be mitigated or prevented; so for this incident we could say that 0.1% of the impact was caused by the initial compromise of that workstation and 99.9% of the losses were caused by all the other failures - so it's effective for any pentest to focus on the latter part, ignore the social engineering and phishing part (since you can't really eliminate it anyway - unlike many other flaws) and assume initial compromise as a starting point.

Pen testers generally won’t physically assault your employees, bring guns and wave them around or shoot people, or kidnap their family members and torture them until they do what they want. What’s your point? Nothing.

I think my security form said something along the lines of “using personal laptops to access company resources grants implicit permission for my company to audit my laptop.” It goes on to explain that if you don’t agree to it, simply use the company device only.

> responsible employees who accept the fake interview but then use their personal laptop to do the interview will have their personal laptop compromised by the pentesters.

If your laptop is successfully compromised by the pentesters then pretty much by definition you weren't being responsible. Though I agree that pentesters going after personal devices is ethically dubious.

Compromising employee personal devices is strictly out of scope for any security testing I've seen.

> Though I agree that pentesters going after personal devices is ethically dubious.

It's not just ethically dubious. It crosses a lot of legal boundaries. A company can consent to have their personal devices hacked. An employee doesn't automatically consent to let a company's contractors access their personal devices.

fake interviews might be excessive but lots of companies send tempting phishing emails as part of the secops, then both collect org-level stats and educate (or shame depending on your take) the employee for getting duped.

Companies don't try to set up job interviews to test their employees loyalty?

Everyone has their price. It is foolish to think that most people would not leave for better title/pay/job alignment/whatever.

Well also to test that they don't just casually drop corporate secrets...

Sometimes called "Full Spectrum Attack Simulation" where initial entry can include physical access or spear phishing. They can span months in elapsed time (not effort) so as to be more realistic. However there are still ethical and safety constraints and even phishing may not be able to use the (effective) technique of stressing the recipient too much or of course blackmail or coercing. So I think fake interviews may not be allowed in many cases, to protect the employees.

Is that even legal? Wouldn’t a fake job interview be, in a sense, tricking you into taking time off and working without pay?

I think it would be legal but unethical -- unless you also then gave the employee substantial compensation for their time and opportunity cost.

Legality would be very jurisdiction dependent. It might be illegal in most developed Western countries, depending on the exact facts of the case. I suspect it wouldn't be illegal in China — unless the trick happened to hit a ranking Communist Party member.

And you never, ever, ever install software as part of the interview. Decline and hang up if they're asking you to install stuff. Visiting websites - maybe but even that unless it's a well known website I wouldn't go there but download and install stuff, hells no

I fully agree with this.

If you ever meet with Alibaba for any reason they'll ask you to download their conference software. Unfortunately most people in the China ecosystem don't have the guts to say no and will just gleefully install any .exe thrown at them.

Hmm like building an node project, actually any project can have a dependency on a made up libraries that does the exploitation or even the project itself as made infamous by chrome extensions

"fork this repo and fix this bug" is a common job interview request, isn't it?

For many build chains, that involves implicitly installing software.

Such toolchains are dumpster fires whose use invites supply chain attacks. They should never have gained mainstream acceptance and should never be run outside of a sandbox.

That said I wouldn't trust a compiler to be hardened against malicious input and anyway you presumably need to run the output so all of that should be happening in a hardened sandbox regardless.

It'll be hard to time a job opening receiving interviews to line up with intrusion tests in a way that people would not be on guard against it in a way not normal.

Fake interviews would be a huge time sink and probably piss off your staff.

They don't have to respond. It gives employers an even stronger signal they should be kept right?

Using pentesting to also test employee loyalty is a great way to get the entire company to hate working there.

Which is why it's going to happen.

Only if your manager is an incompetent moron.

Which is also why we, collectively, as an industry needs to not put up with that. It's easy for me to say from the comfort of it not being my job on the line, that if this happens to anybody they should quit and then name and shame the company, but a competent security team is going to have the idea of doing a fake phishing interview and then not do it because of how morale destroying that would be. The security awareness team sending fake phishing emails are embarrassing enough.

Yes. The situation improves considerably if, when making server changes or retrieving data, programmers are required to enter a code generated from a hardware TOTP token isolated from both PCs and smartphones.

Every service provider system should be designed with the assumption that programmers' PCs and smartphones may be compromised.

What if I plant a hook on the engineer’s laptop that will slightly modify their commits in such a way that eventually I smuggle malicious code over? First, exfiltrate the code over to me, learn from previous commits, start poisoning. Stamped by the CI, no less. In the days of slop PRs thousands of lines each shouldn’t be even too hard.

The point is that at some point you do need to open the door and let the code in.

Wouldn’t most companies worth exploiting run some static analysis in the ci cd

The company should require both automated and human review of PRs. Mine does.

This is the standard at most professional companies but it's now becoming a problem due to AI and undisciplined people submitting 10,000 lines MRs.

mine mandates no security bugs in PR :)

"too long, LGTM, now buzz off"

Well, that's still a human review. If something breaks, you (as in: this reviewer) is at least partly responsible.

shouldn't we focus on a unregulated monetrary system whose sole existence appears to be to support the Epstein class against democracy?

Reply on news.ycombinator.com