Hacker News Reader

> It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion.

not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.

I see a lot of saying that, but no evidence.

A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration?

The answer to that is not obvious.

(edit: response specifically regarding "astronomical" costs)

What makes you think that a multirepo system would be cheaper to maintain than a monorepo? It's just moving complexity from one place (source control software) to another (everything that interacts with the source control software). TANSTAAFL

I never argued that I thought multirepo would be cheaper than monorepo. Or vice versa, for that matter.

I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument.

TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate.

I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty.

Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture.

Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice.

I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center.

Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time?

Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence.

If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards?

Yeah, I'm going to call bullshit.

I think you are looking at this from a hindsight perfectionist bias.

Maybe that is the right decision. But Google has organically grown over the decades and their tooling evolved with that.

It's hard to say which requirements they were working with and what they were trying to optimize for - sometimes higher costs are just okay if they allow or enable other things.

There is validity to that, it just seems to be a very odd decision to me. If I was going to do custom infrastructure, Source control is not where I would be looking. I'd rather put up with a few warts there and focus the time and resources that saves elsewhere.

For me it's always "what are we trading?". In an organization the size of Google, it's far easier to amortize away the cost of such a thing. That doesn't mean it's not a cost, though. And there's a lot of tooling around modern SCM that they won't get because they're on a bespoke system, so there's a cost there too.

As you say, at the time the variables may have been very different.

It's just an area where if I had to make that decision (even back then), someone would have had to be exceptionally persuasive to convince me to authorize what is effectively a multi-million dollar project... to recreate source control. =)

You're comparing "expensive custom solution" with "free off-the-shelf solution" when in reality it's "expensive custom solution" vs "expensive off-the-shelf solution". Source control at Google scale is difficult and you can't just wave a magic wand and make that complexity go away.

> what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?

Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.

So I was at FB when they attempted to add support to Git for really large monorepos. They offered cash and people to work on it, but the git team was completely uninterested and basically said that you're holding it wrong.

Perhaps Google tried to do it this way and hit the same issues.

Git also just doesn't play nice with monorepos of the scale of Google's/Facebooks. From a different article explaining why FB moved away from Git:

"The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1]

[1] https://graphite.dev/blog/why-facebook-doesnt-use-git

Yeah, I remember this. The team involved did try really hard to get the Git team to listen to them, before working with Mercurial (and later building their own thing, although that was after I left).

I don't think you understand how much early Google values engineer productivity.

> Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.

God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool.

Git sucks so much compared to Piper that it boggles the mind to even ask such a question.

I haven't used piper, so I can't comment on that. But people seem to have focused on git. Remind me not to actually use product names as examples; it inevitably triggers people into holy wars and prevents them from actually seeing the point.

I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know.

The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency.

I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue.

It's always a trade; those dollars can potentially do more good somewhere else.

You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH.

It's really hard to deny that Google has acute NIH.

Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions.

I don't think there's anything wrong with that.

I don't think you are wrong; it's just that a lot of people here on this thread are current or former Googlers. Compared to them, you don't know Piper so your argument is at a disadvantage.

My perspective is simple: inventing its own source control system at Google is absolutely worthwhile and the ROI is highly positive. The tradeoff is worthwhile. You might say that before Piper existed it might be a gamble whether it would be worth it. But in hindsight, absolutely 100% yes it's worthwhile. It does not need to be argued, for anyone who has used it.

The only thing that is really subject to debate is whether Google's engineering leads had the foresight of what Piper would accomplish or whether they simply wanted to build something custom due to NIH and it only fortuitously turned out to be good. That's a question no one here can answer.

> I don't think you are wrong; it's just that a lot of people here on this thread are current or former Googlers. Compared to them, you don't know Piper so your argument is at a disadvantage.

Bah. Text boxes in browsers suck, and mine's not cooperating at the moment, so instead of writing another book, I'll just say: thank you for the reasonable response. =)