> Fetched naively, that would be over 600 PB of git traffic per day.
This seems like an absurd amount of data / traffic per day. Is there really that much to code that needs changing on a daily basis?
This seems like an absurd amount of data / traffic per day. Is there really that much to code that needs changing on a daily basis?
As the article explains, "over 600 PB" = 25M code checkouts * 25 GB "compressed working copy".
So "fetched naively" here means doing a full `git clone` on every need for `git checkout`, which of course nobody should do, nor does that refer to "code that needs changing daily".
It can be cheaper to do a full clone than a shallow one, because with the shallow one, you can't re-use packfiles. GitHub has fairly regularly asked people to stop doing shallow clones because it's harder on their servers than full ones.
At some point you still end up running into the issues the article talks about, all of these approaches only get you so far.
There are lots of cases of "we wrote our own X, here's how it works", but rarely the concrete internal knowledge shared about what existing Xs can't do and why.
When the motivation stops at "look, the naive thing wouldn't work" even before the "normal" thing everybody does (fetch), it leaves many open questions to which you probably have an answer already.
We don't do it this way, for obvious reasons, but it means that there are different levels of caching and infrastructure required to make git work at repo sizes far smaller than what we have.