It's expected. Cargo has very conservative cache cleanup (once per month IIRC).
There's incremental compilation cache that generates lots of churn (although Rust's compilation unit is a crate, "queries" in the compiler are cached individually, creating many cache entries per function per type).
The other half of the disk space is debug info, because abstractions that optimize down to zero runtime code still carry all of their source-level debug info.
These can be adjusted in build profiles if you're ok with coarser debug info or less fine-grained caching.