Is this legal? I know I can’t try and break into my neighbors house even if I have no intent of going inside and stealing once I break the lock.
They didn't break in. They found a key that their neighbor dropped and returned it.
> Is this legal?
Generally, yes (though ask a lawyer if you're going to do security work). Security researchers do occasionally get legal flak though, depending on which idiot they annoy by pointing out issues.
Look at pages 10-17 to see how the law is evolving here.
From that lens, that feels like users could be held liable for what these hacking agents are doing. Which in some cases probably makes sense, but certainly not all.
I've seen plenty of cases of Claude having an action blocked so trying tons of workarounds to accomplish its goal, I could easily see it doing this on something more broad.
The service that this whole thread is about is explicitly a "hacking agent", designed explicitly to try to hack things, and was then pointed at a third-party (seemingly without their permission).
Anthropic/OpenAI can reasonably claim that they had no intent and are trying to stop it. OP here did this explicitly and purposely.
In terms of "We wrote a hacking agent designed only for hacking and sell it as a self-hacking service and then pointing it at someone else and omg can you believe what it did we had no intention of hacking" sense, I don't think that's really applicable.
The mens rea is explicitly there and it's not valid for them to try to hide behind an "agent".
Ya, returned it after poking through all of the drawers and iterating through business information that they found.
There is a white-hat line that OP very clearly crossed here.
What's interesting to me as someone who has sold a lot of software to a lot of software companies is that many enterprise vendor agreements explicitly allow companies to pentest their vendors with advance notice and coordination. I don't think any of our clients ever exercised that clause; I expect it's going to be exercised a lot more going forward because it's so easy to do now.
https://www.nilsbecker.de/rechtliche-grauzonen-fuer-ethische...
See also the German Criminal Code, starting with §202a "Data espionage":
https://www.gesetze-im-internet.de/englisch_stgb/englisch_st...
What did I miss they did that's illegal? It looked like it downloaded a public docker image, searched around inside, and verified that the key it found was still valid (without making any changes), and then immediately notified them about the issue.
The white-hat line stops at "validated the key was valid". It does not extend to "poking around inside to extract business-confidential customer information".
Announcing that their agent restrained itself even though it got hold of a live token is necessary to convince prospective clients. You don't want a pentester that doesn't show this kind of reserve!
I suppose so, but with a few words it would have been totally unambiguous though. "So... we pointed Strix at .baseten.co and let it run without credentials or source code (with Baseten's prior authorization, of course)*."
We're in the know about this industry convention, but Strix's prospects may not be.
> You don't want a pentester that doesn't show this kind of reserve!
Agreed! A long while back a prospective acquirer set their red team on the B2B I worked at during due-diligence (with our knowledge). I'm ashamed to say that due to a swiss-cheese-type failure in a very obscure endpoint they eventually gained broad access and exfiltrated our tenant DB. We detected this, and patched the problem, locking them out. The game was well and truly over for us at that point, and we took the loss, but they proceeded to attempt to crack customer credentials to re-infiltrate, causing an emergency that we were then bound to notify all of our customers about--they were damaging the goods! All they had to do was show us a tenant slug list and we would have known the scope of the breach, no further penetration was necessary. The acquisition did eventually go through. Though a highly capable red team they were, I haven't worked with one so reckless since then.
- AI Richard Nixon