Cargo performs dependency resolution, so fetching from git would (and does) significantly slow it down - it wound need to perform git operations to get the tags for each dependency (quick) and then checkout each tag to look for the subdependencies of that package at that version (slow).

Having a central registry is a must for package managers that perform version resolution and want to do so quickly, as it can serve them all the metadata they need to do that resolution.