I don't see the problem. Since it's an external crate, it will compile exactly once after adding it to Cargo.toml, and will be cached forever.
I also had a project a couple of years ago where I had to compile it in release mode to get reasonable performance for testing (Iw as processing GB's of data), which meant 2-3min incremental compile times. Not fun!