Except that it is often a memory-safety problem that enables an attacker to make a program misbehave, through things like buffer overflows. A memory-safe program is much harder to exploit.
Even with a safe language, there's vulnerabilities like supply chain attacks which allow malicious code to use an escape hatch to access memory outside of the process. I.e. I could be programming in Rust, but one of the crates I depend on could silently add an unsafe block which does nefarious things. OS/hardware level sandboxing could prevent many such classes of exploits.
The problem is not about memory outside of the sandbox, but inside. Please read about return-oriented programming, for example, where a buffer overflow bug of a process can be exploited to hijack said process to do work it was not meant to do normally. If this error happened for example in sudo, it can very well be used to gain privileges and do real harm — and this whole problem domain is almost entirely solved by using a “safe” language.