Have you tried sccache [0]? It doesn’t always choose to cache a dependency, but it helps about 70% of the time. Anecdotally, it hastened a release build of a pretty standard CLI tool (with incremental compilation) by almost 4x.
In the context of resource-constrained machines, one can always host it remotely on S3. (or mount an NFS share as the CARGO_TARGET_DIR, if you’re feeling adventurous or want fast CI)