I had OpenSSL as a dependency, and if `PKG_CONFIG_PATH` changes it rebuilds it (this is correct), which means if you make an edit to a file and save it, rust-analyzer would blow away the cache, then you build it on the command line and it blows it away again.
To test:
1. If you quit VSCode and do an incremental build is it still slow? 2. Try `CARGO_LOG=cargo::core::compiler::fingerprint=info cargo build` (took me a while to find that; there is a bug open to make it less stupidly hard to find).
That will print a load of info about why Cargo is rebuilding stuff. Note that it isn't really in a sensible order - the first message isn't necessarily the cause. The message about the environment variable changing was somewhere in the middle for me, so read the whole log.