I think most debug info is in its own section. On a real OS the pages are loaded lazily so no (significant) performance hit.

Still, I hate big Rust binaries as much as the next guy.