I feel like the problem is that it isn't just a package manager... it insists on simultaneously being not only "a" but "the" build tooling, which is trading way too much flexibility for a very small amount of convenience. And like, OK: maybe this is useful for people who are just getting started, but I would firmly argue that this kind of tooling should consider the needs of the expert and not make severe tradeoffs for the needs of the beginner unless it is trivially and obviously optional and there can be growth through changes in tooling (where maybe people "graduate" from Cargo to a more extensible tool... but since it is also the package manager, that's brutal).
The issues for the tool are then riddled with bugs that have been open for years (I think some of the ones that I've been blocking on might already be at 5 years, which is shocking to me as I don't even think of it being very old) that even often have multiple reasonable pull requests filed but where no progress ever gets made on fixing the problems because, even though it is often trivially shown that there is no way to use the tools correctly currently in the general case, they insist that the behavior of a working tool might break some theoretical user relying on the current inconsistent set of behaviors.
The result of all of this is that there are a ton of workarounds for Cargo that projects have started to have to embed almost as boilerplate into their projects, including a popular one that is going around right now (required for Linux->Linux cross) involving setting an environment variable that explicitly tells the user not to set it ever in the name of the variable but like... Cargo doesn't be have any other way of making this stuff work correctly. It is honestly infuriating and ridiculous and there are good reason why you keep seeing stuff like Google avoiding it entirely as part of their Bazel integration.