4,753 karma · joined September 16, 2022
Rapid7 -> Dropbox -> Grapl -> Datadog
This is why I'm building this language. It's capabilities based. Build scripts can't just do whatever the hell they want to, everything is auditable, and it layers its sandboxing.
It's for fun and anyone looking should understand that this is AI driven building with human driven design, but the goal is to demonstrate.
> The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down
Yes, but you can... mitigate the risks? I've explained this.
> The other flaw in your plan is that people make mistakes, a lot of them, all time
Yes, you mitigate the risks. That's why you layer things.
> I spend $x on security won't save you
No one is saying this.
> AI already hacked Hugging Face with brand new zero days like it was nothing.
No, it cost OpenAI money, and those zero days are unsurprising and probably are like ~O(10K) at human level.
> The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.
Science fiction and not supported. The vulnerabilities found by AI are not surprising in the slightest.
I've made my point abundantly clear.
Rereading this, it's an overstatement. It doesn't "virtually eliminate" it. It drives the cost up by like 3 orders of magnitude and it pairs well with other mitigations. gVisor escapes can and will happen, I highly recommend you do more than just stuff something into gvisor - additional seccomp can go a long way, and apparmor/selinux pair incredibly well.
This is obvious. AI doesn't drive the cost to zero and exploitation has always been about cost. Tokens cost money, not everyone has $10M to burn on chaining bespoke 0days.
Consider that if the cost of exploitation was truly 0, then the cost of perfect software would be 0 because you could exhaust an infinite search space of vulnerabilities for 0 cost. Your conclusion could never follow from your premise.
Yeah but that's a business decision. I work on security at a company that does sandboxing and when the company decided to build an AI harness I was brought in as one of the earliest engineers on the product. We do almost everything on that list and we're a fraction of the size of OpenAI. And it wasn't particularly hard, and we have harder requirements imo (because we solve more general problems vs "run a very specific agent with a very specific task and very specific access").
> And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.
Very little software is incompatible with running in gvisor, for example. Most people can just overwrite `runc` with `runsc` and things will "just work".
Running an artifact repository in isolation isn't particularly novel or complex either. You can virtually just eliminate SSRF vulnerabilities with a host based firewall or AWS Security Group etc, like the whole problem goes away by just saying "this box can only talk to that box".
Tools like Smokescreen exist, they work great, they're super easy to deploy. I bet OpenAI could do it, I bet they could run 500k tokens just fuzzing and eval'ing it for 0days for like 48 hours before they actually deploy it too.
OpenAI as a business chose to not bring people who know these things in, or didn't empower them, or didn't prioritize it organizationally. I'm not a genius for saying "use gvisor, set up a firewall, isolate resources" - I'm quite sure there are people over there who would get it done in a weekend. But they didn't, and that's notable.
This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device.
Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user".
Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days.
So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker.
Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does.
NOW you can try to attack the proxy by scraping its memory or whatever.
This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever.
But one thing is certain - it is never "one crack".
This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws.
I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction.
Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
Yeah, basically. I mean I'm handwaving but yes, some combination of those would have made the attack way too expensive.
I could go on and on. A tiny bit of forethought and effort pays off massively.
I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages without decades of serialization vulns have existed, capability-safe libraries have existed, iframe sandboxing, trusted types, content security policy, network ACLs, isolating proxies, fuzzers, formal verification, refinement types, etc.
It's crazy just how safe software can be if you put the effort in. With AI I think we're just seeing how little anyone has bothered to leverage this tech.
OpenAI put shared JFrogy infrastructure in front of their sandbox. I mean, really? Whipping up a hardened artifact infra project with AI is trivial these days and it could have had 1% of the attack surface, been totally network isolated, totally infra isolated, fuzzed, sandboxed, etc. Why didn't they? Stuff like this feels inexcusable for a company with effectively unlimited tokens. I've literally done this with a "pro" subscription.
Show me an AI that breaks out of gvisor wrapped in Firecracker with an credential-injecting proxy and real network isolation. We already know that Mythos couldn't do it - the vulnerability it found in Firecracker required incredible effort and positioning just to not be exploitable. I'm not saying there are zero vulns in it, but the cost is insane.
It's INSANE to me that OpenAI has to say "we now use proper sandboxing". To be frank, it's a bit disgusting to me. I've recently built an AI sandbox and gvisor was just the start of that conversation. If I were OpenAI training hostile models I'd probably start with gvisor, harden further, and potentially consider the entire piece of hardware compromised - they can afford this, they could reflash firmware after evals etc.
I have no clue what you're talking about and I doubt that it's relevant.
The claim was made that this is "simple". It doesn't feel simple to me. I'm asking about the criteria used and how this fits into that to justify the "simple" claim.
The implementation details of this construct are implementation details of the language.
Of course, they don't have to learn anything other than "use `go vet`, it's what enforces this" to use it. But that's true of almost anything?
This feels like at least two layers of special behaviors.
> so management has to step in to baby us.
An engineer wrote that post lol
I could just as easily say "webscale" is great because I can write a paragraph about how it will address all of our scaling problems and carry us into the new decade of infrastructure or whatever the hell.
These paragraphs of "it's good" all sound right in isolation but they're purely negative for actual conversations. Actual technical conversations require understanding what problems have to be solved, under what constraints, and then determining properties within the solution space. I am shocked at how controversial "map requirements to solutions" is, how desperately people seem to want to turn that into a one word conversation as if it's onerous.
I've exclusively seen it used as a thought terminating sentence.
This article is bad. It has led to bad things. It has a bad premise. It has bad ideas.
The correct solution to the problem it wants to address is that engineers should justify technical solutions by mapping product level issues to technical solutions.
That's it.
Why? What if doing 1 new thing saves you from having to do 3 old things? Why 2? Why 3?
> The post is written for an engineer at a startup as a reminder that although it’s green field development, you only have so much runway, so it’s better to focus on what matters instead of trying some new tech because it seems cool.
I am not suggesting that you do something "because it seems cool". I'm suggesting that you evaluate the costs and benefits of technology, and that "boring" is not sufficient nor helpful in that evaluation.
> Ideally. But I’ve worked with plenty of engineers who get far too excited by shiny new tech and overvalue its potential while undervaluing its risk.
I've worked with plenty of engineers who dismiss technology because it is "hyped" etc and then try to build systems on top of databases or other tools that were never designe for the use case and fail miserably.
The issue in both cases is engineers not evaluating solutions correctly. "Boring" will not help. I have literally been in a meeting where "boring" and "innovation" tokens were the justification for a technology choice that failed miserably. More than one!
> Yeah maybe, but unless you’re working on a problem that the tech directly solves, it’s pretty unlikely.
I'm not sure what you mean. There's is presumably some technical solution to the problem you want to solve.
> In 2015, MongoDB was a dumpster fire (which is still kind of true) and node.js was still kind of new and had enough rough edges that most teams were probably better off choosing some other language/framework.
I don't see why you couldn't justify not using Mongo or Node and then determine they aren't fits without saying "they aren't boring enough". In fact, I don't think either of those aren't boring in the sense that document databases/ nosql aren't particularly shocking concepts - the issue was, by far, the implementation.
> “Boring” and “simple” are ways to convey that it’s good to be risk averse.
They are very bad at this.
> It’s a bit of rhetorical flourish
I don't think rhetorical flourish should have much of a place in technical discussions. If you find yourself reaching for rhetorical flourish, you should ask yourself why you can't justify your position on its merits.
> choose what you work on carefully because you have limited runway and should spend that runway working on the problems that matter for your business, not new tech that’s orthogonal to it.
New technology may not be orthogonal to it, it may be critical.
I feel like I've been pretty clear.
1. "Boring" is a bad term that obfuscates things that do not merit obfuscation. It's like saying "brb" in person instead of "I'll be back in 5 minutes" - "brb" means nothing to me, 30 seconds, 5 minutes, 30? And it saved you like 2 words? Silly, unserious, and fine for casual conversations but not for technical decisions like "what database do we use?".
2. It lets people lean into a bias. I see massive bias away from "hyped" tech. People on HN constantly condemn projects for being "hype" tech with no merit behind their criticism, or at least no expressed merit. It sounds very smart to say "oh that's just hyped" - this is a known bias that extends outside of tech. Calling something "boring" makes you sound smart but it doesn't convey information and it leads to the exact same decisions that "hype" would.
Having a conversation about technology is not hard. It's the job. It's one of the most important parts of the job. It is not worth papering over with some biased wording that is guilty of the same exact failure modes of the approach it attempts to deride.
I don't think that boring means any of those things. Most people here don't seem to agree with it meaning those things either.
> If an engineer fails to see the nuance then that's a communication issue and I think they'd probably be a hard person to work with regardless if this article existed.
It seems way harder to work with a group of people who all seem to know their in-group definition of a term and resent the idea of just using explicit descriptions that map.
At work we use Rust primarily for services. It was chosen because we're doing a lot of low level and performance sensitive work. The risk we discussed most was devs not knowing the language, which we decided to hedge against in various ways and accepted that risk.
We use Postgres for a lot of data. We've used it a ton at the company and have a lot of expertise. It handles relational data well, Row Level Security helps us with our multitenancy goals, etc. We discussed some risks, like write load on the db, and have mitigations in place for that that I don't want to get into much.
We use gVisor for isolation. This was a more novel pick for us but we had very strict security requirements and the only two options we considered viable were Firecracker and gVisor - we didn't want to require KVM/ hardware support so we went with gVisor and have been very happy with it. There was a sort of "bake off" to evaluate solutions here.
In every case we simply determined our requirements based on product features we needed and decided what to use. Surely we'll regret making a decision eventually but we've had reasons for these decisions every step of the way.
> Encouraging your team to be selective in where they place their new bets - and use "boring" aka already-understood technology for the bits that are not going to help solve unique problems - can help avoid expensive mistakes.
Or just encourage your team to have rational discussions about technical choices, how is this controversial?