Hacker News Reader

Comment by nh2 | Hacker News Reader

Makes sense. I wasn't suggesting that one should do a shallow clone though. If your company repo is 100 GB, it seems sensible to keep it cached locally on each machine once (100 GB isn't large for current NVMe disks), and `git fetch` the updates the normal way. Then checkouts don't fetch anything large.

View on HN

Ah, yes, you can do stuff like that too, for sure.

At some point you still end up running into the issues the article talks about, all of these approaches only get you so far.

It sounds like it would be simple, but there are a lot of logistical issues with making git happy with this in practice. Repacking things for repo maintenance, for example, but there are many other problems.

You don't actually need to repack the whole repo. Partial repacks are rather easy to implement.

That's true, but logistically speaking the way that git wants data to exist on disk makes it very, very hard to do the kinds of things we needed to do given how many nodes we have that need source code and how quickly the repo is growing. We do have some ideas on how this can be improved upstream!

I'd be interested in a blog post on what those problems are.

There are lots of cases of "we wrote our own X, here's how it works", but rarely the concrete internal knowledge shared about what existing Xs can't do and why.

When the motivation stops at "look, the naive thing wouldn't work" even before the "normal" thing everybody does (fetch), it leaves many open questions to which you probably have an answer already.

It is sort of semi-ironic to suggest this in this thread, but you can ask Claude about why Facebook created Mononoke and Sapling, and where git has scaling issues. It will give you good answers.

Reply on news.ycombinator.com