Hacker News Reader

Comment by the_duke | Hacker News Reader

Very well written post and upstream work is always appreciated.

I also really like monorepos, but Git and GitHub really don't work well at all for them.

On the Git side there is no way to clone only parts of a repo or to limit access by user. All the Git tooling out there, from CLI to the various IDE integrations, are all very ill adjusted to a huge repo with lots of unrelated commits.

On the Github side there is no separation between the different parts of a monorepo in the UI (issues, prs, CI), the workflows, or the permission system. Sure, you can hack something together with labels and custom bots, but it always feels like a hack.

Using Git(hub) for monorepos is really painful in my experience.

There is a reason why Google, Facebook et all have heaps of custom tooling.

View on HN

I really like monorepos. But I find that it's almost never a good idea to hide parts of a source code from developers. And if there's some secret sauce that's so sensitive that only single-digit number of developers in the whole company can access it, then it's probably okay to have a separate repository just for it.

Working in environments where different people have partial access to different parts of the code never felt productive to me -- often, the time to figure out who can take on a task and how to grant all the access might take longer than the task itself.

It's funny that you mention this as if monorepos of course require custom tooling. Google started with off-the-shelf Perforce and that was fine for many years, long after their repo became truly huge. Only when it became monstrously huge did they need custom tools and even then they basically just re-implemented Perforce instead of adopting git concepts. You, too, can just use Perforce. It's even free for up to five users. You won't outgrow its limits until you get about a million engineer-years under your belt.

The reason git doesn't have partial repo cloning is because it was written by people without regard to past experience of software development organizations. It is suited to the radically decentralized group of Linux maintainers. It is likely that your organization much more closely resembles Google or Facebook than Linux. Perforce has had partial checkout since ~always, because that's a pretty obvious requirement when you stop and think about what software development companies do all day.

It’s somewhat mind boggling that no one has made a better Perforce. It has numerous issues and warts. But it’s much closer to what the majority of projects need that Git imho. And for bonus points I can teach an artist/designer how to safely and correctly use Perforce in about 10 minutes.

I’ve been using Git/Hg for years and I still run into the occasional Gitastrophe where I have to Google how to unbreak myself.

git have added recently sparse checkout and there is also Virtual File System for Git from Microsoft.

From my experience git/vcs is not an issue for monorepo. Build, test, automation, deployments, CI/CD are way harder. You will end with a bunch of shell scripts, make files, grunt and a combination of ugly hacks. If you are smart you will adopt something like bazel and have a dedicated tooling team. If you see everything as nail, you will split monorepo into an unmaintainable mess of small repos that slowly rot away.

I have had the exact opposite experience. Long term maintenance in a monorepo is much easier because all developers have visibility into changes. Tooling isn’t really all that complex (at least to me and I’m not a devops engineer either). It’s way more complex to me to have multiple repositories. I’d rather have hundreds of directories that follow uniformity rather than hundreds of git repositories that may not

For me the complexity comes when you have many isolated projects sharing code. With a mono-repo that shared code is immediately consumed by all isolated projects, most of which the developer is not actively looking at to look for breakages.

This means that every single commit every single test suite has to be run across the whole organization. For a small startup that's probably fine, but for a larger organization that adds a ton of issues and extra time. If your change to a shared project breaks another project that you may not be familiar with you end up not only with a delayed context switch (since you don't know about it until the CI has finished running all tests across the whole org), but now you have to figure out how to address it before someone else changes code in that shared project which causes a merge conflict.

Compare that to a split-repo setup where projects are sharing code indirectly (via package management systems). When you change code other projects will only consume the new code when they actually upgrade their packages, and usually a developer that is familiar with the code base will actively hit that breakage in real time.

Granted, this has its own trade offs such as all your systems may be running different versions of shared code, and the delay in breakage detection can cause some back and forth in breakage between isolated projects.

This to me sounds like a break down of documentation and software architecture patterns, not so much a flaw of a monorepo. I’d rather it come up at that stage anyway even if it’s a pain, because it will help re-enforce better patterns and documentation as people feel these pain points. Shared libraries should have a code owner that is responsive for this if it’s not agreed that it’s straightforward enough to work on as an individual dev no?

That’s my take anyway

That why I mentioned Bazel, once monorepo reach certain size you need to invest in right tooling. What people do instead is splitting into multi-repo and keep using the same tooling. By splitting repo you exchange easy problem of scaling tooling for very hard multi-repo/micro-service/coordination. IMO, the biggest problem with splitting repo is Conway Law, once split you change important dynamics of product development.

On this note, GitHub does reach out to customers with Monorepos and is aware of their short comings. I think overtime we will see them change to have better support. It's only a matter of time.

I’ve always found that the biggest issue with monorepos is the build tooling. I can’t get my head around Bazel and other Blaze derivatives enough to extend them to support any interesting case, and Nix has too many usability issues to use productively (and I’ve been in an organization that gave it an earnest shot).

Can you please give an example of such an interesting case? I am genuinely curious.

And I agree with the general point that monorepos require a great build tooling as a match.

An interesting case of what? A monorepo? Issues with Bazel? With Nix?

An interesting case that's hard to support with Bazel.

Note: I don't quite like Bazel; but my take is that Bazel supports too much configurations and options: my workflows are always much simpler. Which is why I am curious to hear what do your workflows require.

It’s been a while since I tried it (maybe 2018?), but Python 3 support straight up didn’t work despite the documentation. It also seemed like adding support for your own toolchain required you to write Java plugins in a very confusing, inheritance-for-it’s-own-sake, raptly-2000s-OOP way.

Got it. I can relate: while Python 3 support is probably fixed now, I agree that Bazel tooling for adding toolchains has been always abysmal. Things somewhat get better (https://docs.bazel.build/versions/master/toolchains.html) since it's now in Skylark, not in Java, but I still don't like it (aka not productive)

Overall, custom toolchains are common (especially, in the embedded world), and a good build system should make that very easy.

With GitHub actions you can quite easily specify a workflow for parts of your repo (simple file path filter https://docs.github.com/en/actions/reference/workflow-syntax...). So you can basically just write one workflow for each project in the monorepo and have only those running where changes occured.

That’s what I’m doing now, but you still have to push artifacts up to a repository and downstream jobs have to pull them down. Further still, managing the boilerplate for GitHub actions is no small task (e.g., you have a dozen Python projects and you want the actions for each to look similar), so to solve for that you have to build some hack that involves generating the Actions YAML before you commit or similar. Not the end of the world, but still far from my ideal state.

>have to build some hack that involves generating the Actions YAML before you commit or similar

You could probably use jsonnet expressions and invoke them from a pre-commit hook.

That seems fairly elegant to me.

I’m not doing json net, but I am generating them effectively from a pre-commit hook. One of the jobs that I generate validates that the actions that were committed are up-to-date. This works well because I don’t have too many dependencies, but if you have deep dependency trees then the builds will end up taking a long time and you end up reinventing Nix or Bazel poorly.

Are nix and Bazel in the same category? I have used nix very little and never used Bazel. But I have used Make and jsonnet pretty often. Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?

> Are nix and Bazel in the same category?

Nix bills itself as a fully reproducible package manager and Bazel as a fully reproducible build tool. In the abstract, both of these reproducibly build software, so that puts them in the same category in general.

> Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?

Yes, and I’m doing something similar although not with Make. The problem is that this home-grown build system doesn’t do incremental builds, so everything will be fully built from source every time, and that can take a while for very deep dependency graphs.

Conceivably your “Actions generator” script could generate only the jobs that need to be run based on what changed since the previous commit, referencing artifacts from the previous commit for those build steps which weren’t invalidated. This is a neat concept, but we’re well on our way to reinventing Nix or Bazel.

Yeah, in my experience it takes X years of experience at a company that is already using a monorepo + blaze-derivative tooling, for the costs of setup and maintenance to make it worth adoption.

That X is mostly a function of how open you are to the idea that maintaining + using a centralized, language-agnostic build graph is a Useful Thing.

But once that X is crossed, I don't think it's possible to happily go back to a non-monorepo environment. :)

FWIW, we've used Pants successfully for years, which has excellent Python support. However, we're in a long-haul migration to Bazel, so you might be better off revisiting Bazel if you want to invest in something that's extremely likely to issue dividends over the next 10 years.

We used Pants as well, and it worked alright for Python. But it’s codebase was a mess and it was also difficult to extend. I get the strong feeling that there is a better way to make a build system—perhaps something like Nix but without so many usability problems—but no one is investing in that right now.

I wouldn't call it painful exactly but I'll be happy when shallow and sparse cloning become rock solid and boring.

Reply on news.ycombinator.com