There are a lot of places where CVEs don't matter. The HFT firm I worked at groaned over Spectre and Meltdown mitigations, because now our CPUs will be slowed down by not using unsafe behavior. Similarly, I bet a lot of video game folks also didn't like them.

If you aren't connecting to the network or running in any sort of privileged mode, the space of attacks that you need to care about is pretty limited.

Ironically, the answer to a lot of unsafe input in Rust is to crash (if your error handling logic didn't catch it), which is something that you likely care about a lot. Most players care a lot less about corruption of the internal state of your video game than they do about crashes.