Trying to optimize parsing binary interpretation is a lost cause.
The problem with pure virtual machines is they're basically meta-macrocode are mappable back to the ISA macrocode without the overhead.
JIT is the way out performance-wise while the boundaries of the VM are what provide portability and safety guarantees.