Most "interesting" programs can perform I/O. Then you run into ambient authority and confused deputies.
Most programs do not need arbitrary access to the file system, and it should be the OS's job to whitelist which files a program has access to. Again, a safe language is not going to save you from bad behavior on the filesystem either. It really only solves the memory problem.
Except that it is often a memory-safety problem that enables an attacker to make a program misbehave, through things like buffer overflows. A memory-safe program is much harder to exploit.
Even with a safe language, there's vulnerabilities like supply chain attacks which allow malicious code to use an escape hatch to access memory outside of the process. I.e. I could be programming in Rust, but one of the crates I depend on could silently add an unsafe block which does nefarious things. OS/hardware level sandboxing could prevent many such classes of exploits.
The problem is not about memory outside of the sandbox, but inside. Please read about return-oriented programming, for example, where a buffer overflow bug of a process can be exploited to hijack said process to do work it was not meant to do normally. If this error happened for example in sudo, it can very well be used to gain privileges and do real harm — and this whole problem domain is almost entirely solved by using a “safe” language.