If you don't need to care about vm, you probably should be using a GC language. If you do, you certainly need to understand vm.
I'm porting this Python library to Rust: https://github.com/Blizzard/s2protocol.
For the shitty, first pass prototype code I've written so far I've seen a ~30-40x speedup compared to the Python implementation and a ~2x speedup compared to a Go implementation written by someone else: https://github.com/icza/s2prot.
That is why I chose Rust over a GC language like Go (E: I also wanted to learn it ofc). It's a lot faster out of the box, even without me having a strong understanding of memory operations.