Comment by wahern | Hacker News Reader

Comment by wahern | Hacker News Reader

Interesting how at the end, after acquiring out-of-bounds write access, that it was easiest to leverage the WebAssembly infrastructure to execute code than to build a ROP chain.

Apparently WebAssembly heap memory storing generated code is not write protected at all. I guess whatever architecture they have for managing typed memory chunks doesn't make it sufficiently easy to manipulate protection bits dynamically, and the WebAssembly folks were content to leave compiled WebAssembly chunks are RWX.

It's unfortunate that this thread hasn't been upvoted. It really drives home the futility of expecting something as complex as V8 to ever be safe enough to sandbox arbitrary code. And it has little to do with the language of the implementation--the same engine re-written in Rust would have been just as susceptible to the two exploits.

I guess it's just too inconvenient to accept reality. I'm still surprised that Cloudflare and AWS have actually convinced themselves they can make such architectures secure[1]. Less shocked that everybody is else is so credulous.

Though, I am shocked that Netflix is so credulous. A recent blog post described how they use AWS Lambda functions to manage their CA private key. But, again, because of how limited KMS is I guess it's too inconvenient to not trust Lambda. KMS really should support asymmetric key operations; it's ridiculous it's still not supported. I guess AWS's KMS "cloud HSM" solution just wouldn't scale (in terms of CPU) if people could do that.

[1] Absolute security is impossible, but as complex as this exploit was it's obviously still far too trivial. It's a totally unwarranted and unreasonable expectation that bad actors are incapable of reading (if not manipulate) co-hosted Lambda or Cloudflare Worker projects. Writing a non-JIT'd engine with a simplified memory architecture is not only feasible, it would be an obvious candidate for formal verification. But the demands for performance are just too strong and so, as always, security takes a back-seat to performance and speed-to-market.

Hi, TLM of the WebAssembly runtime in V8 here.

TLDR: it's asm.js's fault. And yes, complexity.

The reason that WebAssembly JIT code memory is still RMW (for now) is actually really unfortunate. As you might know, V8's JIT code memory for JS is only writable when the application is quiesced (i.e. JS is not running) and the JIT is either finishing a function or the garbage collector is moving JITted code. It's read-execute otherwise. It's never both writable and executable at the same time. We generally refer to this as WX protection (i.e. writeable/executable exclusive).

In the case of WebAssembly, it's asm.js that's the real culprit here. Internally in V8, asm.js code is translated to WebAssembly by a very quick custom-built parser that validates asm.js's type system while parsing and emits WebAssembly bytecode along the way. The WebAssembly bytecode is then fed to the WebAssembly engine for compilation.

Well...not so fast. In order to meet our performance goals for fast asm.js validation and startup, the WebAssembly engine does not do up-front compilation of Wasm code coming from asm.js. Instead, it does on-demand compilation of asm.js code, compiling the Wasm bytecode corresponding to each asm.js function upon first execution. We call this "lazy compilation".

We originally shipped lazy compilation cooperating with the WX mechanism executable for JS code. That is, upon every (lazy) compilation of asm.js code, we'd flip the permission bits on the code space in order to write in the little bit of machine code generated for each function. Problem is, that permission flip is expensive--like really expensive. So expensive that we had to unship WX protection because it made asm.js code unusably slow.

We're working on fixing this, as we are keenly aware of the risk exposure here.

> TLDR: it's asm.js's fault.

> In order to meet our performance goals for fast asm.js validation and startup, the WebAssembly engine does not do up-front compilation of Wasm code coming from asm.js.

It sounds like the fault is how you reached your performance goals rather than asm.js itself.

It's very obvious from context the reference is to the asm.js implementation in V8, I'm not sure how you could read it otherwise ("strongest plausible interpretation" and all that).

> It sounds like the fault is how you reached your performance goals rather than asm.js itself.

The properties of asm.js make it inherently less efficient than Wasm.

This is super-interesting back story, thanks for taking the time to share it.

I'm not sure how asm.js is relevant here. The issue is that function-level lazy compilation makes compilation too hot to allow for an mprotect call, no? Wouldn't a function-level lazy Web Assembly implementation have the same problem?

It seems to me that the solution is the same for both wasm and asm.js: make the unit of compilation larger than the function, so as to amortize the cost of mprotect.

You and I both know that the comments section of HN lacks sufficient space to discuss all the things we thought of / could think of :)

Also, I should have mentioned, but concurrent compilation of Wasm (for incremental tierup) essentially puts the final nail in the WX exclusion coffin. (but we deployed that long after the first reason, lazy compilation for asm.js). The only solution in the long run afaict is out-of-process compilation, which we will explore this year.

Why doesn't it at least do double-mapping if it can't do W^X flipping?

Author here - thanks for this! I was wondering why Wasm was RWX.

My 2c: I think that there will be risk of code injection as long as write_protect_code_memory in Heap is writable. Changing that flag will usually mess things up and crash writing to RX memory (CodeSpaceMemoryModificationScope won't switch to RW), but a well-crafted exploit might be able to get a fresh executable MemoryChunk (which will now be RWX). It's likely complex to exploit, but the incentive is that code injection is more reliable than ROP when targeting multiple builds (unless you build the chain dynamically, which is slow and often painful).

I don't know what you mean by AWS? They don't rely on V8 for isolation, they use KVM among many other layers.

I guess I was wrong. Looks like AWS is already using Firecracker for AWS Lambda: https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...

Although it would be nice for AWS to explicitly say that there's one customer per hosted Lambda environment, and distinct from language-level or process-based sandboxing. From the article above you have to assume that because v1 of Lambda put one customer per EC2 instance as an expediency that it remains the case.

Nonetheless, regarding the Netflix case, nobody should be hosting long-term CA keys in Lambda or any co-hosted virtualized environment framework. These days few reputable companies except maybe Intel even bother arguing that secret keys are safe from side-channel attacks in such environments. The evidence is just too overwhelming, even for the self-deluding "if it's too complex for me to understand then it must be impossible" crowd.

I believe AWS does share customer information within a device, but with a lot of sandboxing below that. You can watch this talk to learn more: https://www.youtube.com/watch?v=QdzV04T_kec

I think the threat model is basically that you'd need a KVM kernel 0-day + the ability to exploit it, so a point of privilege such as outside of the firecracker sandbox.

Reply on news.ycombinator.com