Comment by geofft | Hacker News Reader

Comment by geofft | Hacker News Reader

Parent Full thread

> in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel),

This is easy to check, and the article is right. The Linux kernel does not have tens of thousands of active developers. Over a little more than the last year:

    titan:~/src/linux geofft$ git log v4.8..v4.14 --format='%aE' | sort -u | wc -l
    4857

If you look at just the most recent release cycle:

    titan:~/src/linux geofft$ git log v4.13..v4.14 --format='%aE' | sort -u | wc -l
    1839

Alternatively, if you look over the last ~year, but at people who have more than five commits (chosen arbitrarily):

    titan:~/src/linux geofft$ git log v4.8..v4.14 --format='%aE' | sort | uniq -c | awk '$1 > 5' | wc -l
    1814

A codebase with 20,000 actively working on the code base is at least an order of magnitude more than the Linux kernel. If they're committing patches every single day, probably more than that. I have, last I checked, four commits in Linux: two for a summer internship in 2009 and two doc fixes. That certainly contributes to the scale of Linux the kernel in the sense of "there are so many people working on it!", but not in the sense of being an "active developer" of the git repo. I was fine sending in my patches to a mailing list and hearing back at some arbitrarily later point, not triggering any CI, not asking anyone else to collaborate on my work, etc. That doesn't work if you have 20,000 engineers on the same codebase writing code every day.

> Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size?

Sometimes that's the best way to get things done? The best Debian workflow I've ever worked with (and I've worked with a lot) involved actually committing binary .debs to SVN alongside their source code, because that meant that the SVN revision number was the single source of truth. There wasn't some external artifact system, nor was there a risk of picking up the wrong packages from an apt repo when rebuilding an older SVN revision.

I'm not defending this as pretty. But I will defend this as sane. It got the job done reliably and let me work on actually shipping the product and not yak-shaving workflows.

Good stuff. Came into this thread expecting the top comment to be a know-it-all answer trying to "debunk" the article, as so commonly seen on HN. This is a perfect, well-researched response.

https://news.ycombinator.com/item?id=4693920

Interesting read

I think this is all missing the point. Do they want to store, say, the entirety of the base Windows OS in a single git repo? If that's the case, sure, git isn't a great fit for their use case, and they should either use something else, or find a way (as they're doing?) to change git to fit their needs.

I can't see MS having tens of thousands of developers working on any single component of Windows, so I'm guessing they _do_ want a giant monorepo. If they were to break it up into separate repos per OS component, I doubt they'd have scaling issues (of course, breaking things up introduces coordination and dependency issues).

I can't see it being easy to take a 20 year massive code base and split it up into components that are so isolated enough they can live in different repositories smoothly.

As they say in: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-g...

The first big debate was – how many repos do you have – one for the whole company at one extreme or one for each small component? A big spectrum. Git is proven to work extremely well for a very large number of modest repos so we spent a bunch of time exploring what it would take to factor our large codebases into lots of tenable repos. Hmm. Ever worked in a huge code base for 20 years? Ever tried to go back afterwards and decompose it into small repos? You can guess what we discovered. The code is very hard to decompose. The cost would be very high. The risk from that level of churn would be enormous. And, we really do have scenarios where a single engineer needs to make sweeping changes across a very large swath of code. Trying to coordinate that across hundreds of repos would be very problematic.

> I can't see it being easy to take a 20 year massive code base and split it up into components that are so isolated enough they can live in different repositories smoothly.

I can't see it being desirable either, they already have to deal with outdated external API, surely they don't want outdated internal API. A single repository lets you replace an API throughout the codebase at once and remove the old one entirely, if you start "modularising" then all "internal API" are suddenly external with all that entails.

PM for Git at Microsoft here. We explored splitting it up. It's a 300GB repository and it's been a monorepo for the last 20 years. Splitting it up logically would take a lot of time (to put it mildly) and development would stall while we did it. And once we did, we would have 100 3.5GB repositories? 3500 100MB repositories? Neither of these are particularly appealing and getting changes checked in atomically across multiple repositories is insanely challenging. There's no doubt that we would need to build tooling to make this work for us. (We did actually explore this direction, but ultimately decided that it would be too much work for too poor an experience.)

Instead, we decided to - as you put it - change Git to fit our needs.

What were you using before this?

We were using a mix of tools throughout Microsoft: several teams were using Source Depot, which is an internally developed centralized version control system. It was built to handle large teams, like Windows and Office. It was the precursor to Team Foundation Version Control, which is the centralized version control system available in TFS and VSTS, which is also capable of scaling to large projects, and many organizations within the company were (and are) using that.

GVFS is part of our effort within the company to standardize on a single set of best-of-breed tools, and use the same tools inside Microsoft that we deliver to customers. So we're adopting one engineering system throughout the company and we're moving everybody to Visual Studio Team Services.

It's my understanding that Source Depot was not internally developed but rather was a fork off Perforce which MS purchased a source license to some years back.

True. Source Depot originated as Perforce and is a fork; it had (at one time) a pretty large team of developers working on it and it has been heavily, heavily modified in the many years since.

Changing Git to fit your needs probably just makes this monorepo function on borrowed time. Fixing the symptoms with band-aids will work for now, but at some point, something will probably have to be done about this 300GB god-repo. Any insight in to what/when/if something is going to be done about that?

I'm not the right person to ask about Windows organization, I'm afraid. But I will say that the nice thing about moving to Git is that its lightweight branching makes refactoring much easier.

Does Git on MSWindows use a dentry like cache in user land to speed up filename lookups? Does it use a stat() equiv cache? Is there a reason that such caches weren't put into ntoskrnl.exe? Would you be able to give us a brief list (say top 5) of changes that sped up git on windows? And which ones had the most effect on super sized monoreops. Thanks!

I have seen poor performance of git on Linux in very large repos, so I'm not super convinced that the dentry cache magically makes things better.

In particular, if my repo is big enough, I often don't have the entire tree in memory (because I'm doing other useful things with memory and caches got evicted). core.untrackedCache makes things a little better, but it's still not great.

+1 for the research and the distinction between sanity and purity.

> actually committing binary .debs to SVN alongside their source code, because that meant that the SVN revision number was the single source of truth

This is totally armchair-devops, but it sounds like a more ideal case would be CI that runs based on your commit. In my experience, developers get lazy and are likely to commit binaries that dont match the code or miss some dependencies or steps.

I completely agree with your point, using the tools as intended isn't always the best way to get things done. Video games are an easy example, code is often dependent and intermixed with binary assets and you want those to be versioned together.

If you actually commit binaries to help with the CI, then those binaries should be installed by the CI. I'm not saying it's a good idea to do it this way, but if you really want to for some reason, then that's the way to enforce no steps or binaries are missing. (And if devs forget to add something, the build would just fail)

Apologies if I wasn't clear, that's what I was getting at. Commit the code, CI generates the binaries. I guess you could commit the binaries if you wanted, but a separate caching repo would make things more manageable. You'd still be using the commit to reference the binary. But this is ideal, and there are good, practical reasons doing what he describes.

For games the binary data isn't from code. They're textures and models generated in a package (like Photoshop). The bits of code are often tightly coupled with the textures--like a procedural waterfall that uses still images. You'd need to version both of those together.

I mean, it's certainly true that at least once I found that another developer had updated the .debs and forgotten to update the sources, and I had to attempt to reconstruct what he did because I needed to add another patch on top. So you're absolutely right that some better CI would have helped here.

But for the product we were building, it was important that developers were able to test their .debs before committing to the repo. That means that CI can't build post-commit: you have to have some way to take a local modification, turn it into a .deb, test it and see if it does what you want, and then commit that.

I've thrown something together for my current job that supports both building a package interactively for testing and building one in the CI pipeline post-commit (using a git repo as the source in both cases) but it's definitely more cumbersome, and I haven't figured out quite how I want to fix it.

(Also, yes, games are a more obvious example of this, but there are lots of purists who've never worked in game development who are somehow convinced that assets aren't code and shouldn't be in the code repo. Having not worked in game development myself either, I figured it was safer not to invite that argument.)

Are private branches that the CI can pick up also not a workable solution for your usecase?

That's what I'm doing now at work, and it's sort of silly to git push, wait for CI, notice that it failed, fix, git push, wait for CI, .... in an SVN world, that would have meant one commit in the monorepo (since there's no such thing as temporary branches, and revisions are global across all branches) for every typo I make during development. You could make it work with git, certainly, but on balance the check-in-a-.deb approach worked fine and didn't require spending more time writing tooling.

You could try gerrit + e.g. Jenkins. When pushing to review the CI can pick up the commit, build it and optionally run tests and give +1/-1 verified.

The commit is merged only after a developer will give +2 for the review.

Question: what's the `sort -u` for? You're piping to `wc -l`.

"u" stands for unique, so after sorting it skips identical rows

Oh! TIL, thanks!!

Dude teach me how to pipe liKe you

Reply on news.ycombinator.com