You need a sandbox but I don't think web-browser sandboxes are designed for your use case, when the web application doesn't trust its own code.

Considering the constraints I would experiment with a virtual machine. It can use your own set of instructions, like high level domain specific instructions in a json, or something lower level like a simple assembly.

You could use qemu in WebAssembly in a webworker and hope for no vm escape hack. Or a lua interpreter implemented in javascript.