Databricks recently rebuilt Spark in C++ in a project called "Delta Engine" to overcome the JVM limitations you're pointing out. You're right, Rust is a great way to sidestep the dreaded JVM GC pauses.
I would love to have something more resource efficient than Spark on JVM, but Delta Engine isn't there yet.
https://blogs.oracle.com/javamagazine/understanding-the-jdks...
One can do both.