Lua has debug hooks that can be used for the purpose; for example an instruction counter coupled with a pool allocator should get one quite far. I would never trust third party bytecode, only source code.
Is anyone familiar with Roblox luau security features?