tptacek

Both, maybe? I mean: whether or not they broke the jailer, we know the jailer is breakable; it wouldn't really even be newsworthy to have an LPE that breaks the Firecracker jailer.

Hey, I don't write AI blog posts. But I want to know how KVM escapes work, too.

I just don't think "can you run CIFS easily under it" has much to do with whether someone like Replit should use Docker, Firecracker, or gVisor.

I mean, it's good to want things, but I don't know what that gets you today. They're not going to give you the hand-crafted post you're asking for.

You can't insist your way through not learning about one of the most important vulnerabilities announced in the last 50 days.

Who cares? The overwhelming majority of cloud customers just want to run straightforward Node.js apps.

I read your whole post, which ended with "if users copy around fingerprints", which is a problem the built-in SSH PKI that large fleet operators already use solves. I don't think I misconstrued you.

You don't have to keep replying, but I'm going to point out that we have gotten no closer to finding a reason that anyone would even entertain the idea of SSHFP, a system that wires the already perfectly good SSH PKI subsystem into the state-controlled DNS PKI. Why would anybody ever do that?

No.

It's very much a Firecracker thing. KVM is part of the Firecracker stack, and getting code execution behind the jailer is still effectively game-over; it's just a somewhat higher degree-of-difficulty game-over. It's a drastically worse place than you would have been last year, when there wasn't a known plausible way to get even that far.

This is a really good bug and nobody should downplay it.

Ah, yes. What even is a "DNS server"? Clearly it's not the thing that shows up in your network configuration under "DNS server"; that, as you observe, is properly labeled "resolver".

I haven't worked at Fly.io for months, during the entire time I worked there I never saw us provide to law enforcement any data that we weren't legally obligated to provide (all of it extremely boring), all such requests routed through our (annoying, expensive) external legal, and we had no facility we could have used to provide "SSH session logs".

The only interesting thing I can tell you about interfacing with law enforcement is that it's pretty funny to authenticate law enforcement people (they can't just give you a phone number to call them at).

I have no idea what this is about. Sorry, you're not going to get any tea from me on it!

No, that's pretty overwritten. The simple story is: like most neocloud providers, they were using Docker and/or Linux containers as a workload isolation primitive, those have never been safe, and they've now worked out how to scale actual KVM isolation to their users.

A thing I wish more people would do: ask prospective cloud hosting companies "is there a mode of this offering where my workloads might share a kernel with other tenants I don't trust, as would be the case with Docker?", and if the answer to that question is ever "yes", disqualify the provider.

Oh, I absolutely have it as a goal! It just shouldn't be possible for me to singlehandedly succeed at it.

Google [if-by-whiskey].

You don't have to distribute fingerprints by hand. SSH already has a built-in PKI that handles this, and it's what most large fleet operators use (it's also the simplest way to connect an SSH fleet to an SSO IdP). SSH certificates, not WebPKI certificates.

I've built systems out of both and they are quite similar, using different mechanisms as vectors towards the same outcome, which is to minimize shared kernel attack surface in untrusted workloads.

Firecracker doesn't require bare metal, but (1) you want it for performance and efficiency at scale and (2) you take a security hit for nesting.

It's a super good bug! I just don't follow the exploitation details all the way. I know more than one cloud provider panic-bounced their fleet in response to it, and I think that was the right call.

Just for what it's worth, that escape is much more situational than PwnEscape (there have been other escapes that depended on nested KVM, as well). Pretty much everyone seems to have been exposed with PwnEscape, because everyone using Firecracker is also using virtio-net.

Wait, this is also presumably true, come to think of it. I'm a little fuzzy because they document the memory disclosure primitive carefully but not the memory writes they also got from it --- but whatever those are, they're hitting kernel memory? But the exploit they're describing here gets code execution out of Firecracker itself, not the kernel.

Yeah, all I'm saying is that I agree with you about Xen, but that I don't agree about the centrality of formal methods --- at least on current projects with current verifications, things could get different now that AI lets us be more ambitious with verification --- in determining attack surface.

Generally, the less your OS can do, apart from strictly hypervising guest operating systems, the safer you probably are. Hence why people do things like seL4.

That's basically what the Firecracker jailer does. Always my thoughts turn to the waitid(2) vulnerability; for a little while, you could pass kernel addresses through it and whack arbitrary kernel memory, and nobody was ever going to seccomp "waitid" out. But yes: it's a meaningful defense, if not one you'd want to bet your service on.

You're responding to a comment where I wrote that I would never use Xen with an argument not to use Xen. All I'm saying is that "formal verification", at least as it exists in current systems, is not why I wouldn't use Xen.

As another commenter noted, the PoC in this article doesn't defeat the Firecracker jailer, which does essentially what a "zone" would do for you. In both cases, that's cold comfort, because even in an unprivileged sandbox environment you still have a huge kernel attack surface to contend with LPEs in.

That's a good catch. Yes, since they've only proven this up to the jailer boundary, you'd need a second and relatively powerful LPE vulnerability to get real host control out of this.

Like some of what I'm saying is just that if you tell a systems security person "we're using seL4 because of its formal verification", you're going to get eye rolls. Whatever else you want to say about it, the formally-verified configuration of seL4 doesn't actually do much, meaning whatever system you're building on top of it is going to be where all the security drama is anyways.

I'm certainly not telling you to use Xen. I wouldn't use Xen for anything untrusted.

The general vibe I hear from security practitioners is that they (gvisor and Firecracker/Cloud-Hypervisor) are comparable. You generally do Firecracker if you can get away with it, because it's simpler and faster, but to do it well you generally need bare-metal hosts.

I think "hardware" vs. "software" is the wrong way to frame this. The big difference is between shared-kernel multitenant and guest kernels. gVisor (in some configurations) is like an 80% solution; a partially emulated guest kernel. I think the verdict is in on shared-kernel vs. not: don't share kernels between tenants.

The hardware isolation itself --- AMD-V, VT-X, &c --- has held up for almost 2 decades now. You never know, of course, but if you were going to place a bet it probably wouldn't be on hardware isolation failing before software did.

It's a settled enough part of HN that you'd do better emailing hn@ycombinator.com about it than you will litigating it in public. I don't care either way, I'm just explaining the behavior to you.

Basically everyone in cloud-land uses it at this point.

You need something more interesting to say than this, because this is an article by the discoverers of this vulnerability, the first general-purpose Firecracker escape that doesn't depend on nested virtualization in recent memory, along with in-depth discussion of exploitability. It's inherently interesting and on-topic and there isn't an alternate source for the same content.

Additional color: per the commit, the bug was found by Paulos Yibelo, who runs PwnAI (the author of this piece), and worked on security at AWS for 4 years prior.