say you have a structure like: projectA projectB sharedUtils
Each time you push you might have a build for projectA and projectB but it builds both each time you push to master. Ideally you could use Git to see if anything in projectA or sharedUtils changed to trigger projectA's build and same for projectB, but I'm curious what others are doing.
However... I would strongly advise not going for a monorepo. No, I don't mean something like tensorflow where you have a bunch of related tools and projects in a single repo. I mean one repo for the entire org where totally unrelated projects live.
Every company I've been at that used a monorepo found themselves struggling to make it work since you need a ton of full time engineers just to keep things working and scaling. Many of the problems that monorepos try to solve (simplifying dependency and version management) are traded for 10x as many problems and many of them are hard (incremental builds, dependency resolution).
Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted.
My contrasting anecdotal experience is that whether at BigCo or on a small team monorepo is almost always the right answer until your requirements get exotic enough that you're in special-case land anyways (like a separate repo for machine-initiated commits, or something that's security-sensitive enough to wall off some contributors).
Both `git` and `hg` scale easily to to really big projects if you're storing text in them (at FB our C++ code was in a `git` monorepo on stock software until like 2014 or something before it started bogging down, I'll gloss over the numbers but, big): the monorepo-scaling argument is brought out a lot but rarely quantified.
The multi-repo problem that gets you is dependency management, which in the general case requires a SAT-solver (https://research.swtch.com/version-sat), but of course you don't have a SAT-solver in your build script for your small-to-medium organization, so you get some half-assed thing like what `pip` and `npm` do.
Again purely anecdotal, but in my personal experience multi-repo too often gets pushed by folks who want to make their own rules for part of the codebase ("the braces go here"), push an agenda around unnecessary RPCs, or both. That's not true of all cases of course, but it's a common enough antipattern to be memorable.
It’s funny, I’ve heard this exact same argument for why you should not use micro services.
At my job our cloud team uses multiple separate repositories which makes sense, but it also moves the burden of versioning to run-time. This is because they have to interface with multiple different versions of the device firmware. So they deploy different run-time versions of the APIs to support legacy and current production firmware versions. But our firmware repository is a monorepo in that the sources and build system builds the artifacts for multiple devices from the same source tree.
So it's not so cut and dried as "never use a monorepo" or "always use a monorepo." It involves engineering tradeoffs and decisions that are made in a context, and you can't extract your advice from the context in which it exists. What works for our cloud team would be a terrible mess on the embedded side simply because of how the software is deployed and managed.
Working effectively in a monorepo does require suitable tooling (full disclosure: I am a core contributor to Pants, a monorepo build system with, currently, a Python focus). But with that tooling in place a monorepo can make it a lot easier to collaborate, to manage changes and dependencies, and to avoid balkanization and fiefs. Granted there are no silver bullets, but we are working to make tooling good enough to avoid needing the large in-house support team you allude to.
Having a large number of small repos can work if they don't have a lot of interdependencies, and maybe this is what you mean by "totally unrelated projects". But if there are significant interdependencies (and repos usually end up this way, in my experience, which may or may not be representative) things become very brittle, and it's very hard to make changes and reason about how they affect downstream code, not to mention the challenge of version resolution, aka "dependency hell".
In the end, managing a large codebase, whether it follows a monorepo or multi-repo architecture, requires suitable tooling. On balance I think that monorepo is often a better choice, but YMMV for sure.
See this documentation page: https://docs.github.com/en/actions/reference/workflow-syntax...
Their example is:
on:
push:
paths:
- '\*.js'
This is worth a read: https://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-y...
The main point is the "better tooling than average" section near the end - you need tools that can understand relationships between different parts of the codebases, and test appropriately, which is where the integration process for monorepos starts to fall apart.
Building on that dependency graph and soundness in Yarn 2, I built yarn.build [0] which works in a similar way to all the monorepo build tools.
It builds your packages based on their dependency graph, skips whats already built, and has a bundle command for creating a zip ready for lambda/docker/etc
If you do a full build on every commit, it gets slow much sooner than you'd expect, and people are going to do less work while they context switch to posting to HN while waiting for their 15 minute build for a 1 line code change.
I worked at Google and we had a monorepo, and there were hundreds if not thousands of engineers working on build speed and developer productivity, and it was still significantly slower to "bazel run my-go-binary" versus "go run cmd/my-go-binary". In many cases, it was worth it, but in very isolated applications, it was definitely not worth it. (And people did work around it, by just setting up Git somewhere and using Makefiles or whatever, and that ended up being even worse. But it gets worse incrementally over time, and you're kind of the frog getting boiled alive.)
Where I'm going with is to advise you to be very careful. The tools to support real productivity in a monorepo are expensive in terms of your org's time. If you can get by with a repo per app and a common modules repo, and just update the app to refer to a version of the modules repo as though it's some random open source project you depend on, you're going to get much farther with much less tooling work than you would with a monorepo. But, the modules repo is going to break apps without knowing, and that's going to be a pain. Monorepos do exist for a good reason.
(The other thing I like about monorepos is that you do less per-project setup work. Want to make some new app? You can just start writing it, and you get the build, deploy, framework, etc. for free. It can be very productive if you're finding yourself starting new projects regularly. In my spare time, I write a software, and I really regret splitting it up into multiple projects. But, it's kind of necessary for open source stuff -- people don't want to download ekglue if they want to just run jlog. So I split them, but it costs me my valuable free time to do something I've already done ;)
My TL;DR is that you will be tempted to take shortcuts and the shortcuts will suck. If your project has the resources to have someone set up Bazel, distribute the right version of Bazel and the JRE to developer workstations, setup CI that is aware of Bazel artifact caching, and SREs to be around 24/7 to support your now-custom build environment, you will have a good experience. Be aware that a monorepo is that level of investment.
Meanwhile, if you just have a frontend and a backend in the same repo, you can probably get away with a full build for every commit. And you don't need that shadow team of tooling engineers to make it work, you just need a docker build, and a script that runs "go test ./... && npm test" or whatever ;)
This ordering is probably in order from "most" to "least" obvious, and I don't have an answer for what the correct solution is. Most probably "separate repo", making the interface not privileged to either side of the "interface barrier".
FWIW, building at Google when at a 8h offset from Pacific Time worked fine, indicating that part of the problem is "does your repo/build solution scale to tens of thousands of simultaneous users".
I have a big monorepo at work but whenever anything changes I want to rebuild everything to generate a new firmware image. I have ccache setup to speedup the process given that obviously only a tiny fraction of the code actually needs to be rebuilt.
It's a bit wasteful, sure, but if I were to optimize it I'd be worried about ending up with buggy, non-reproducible builds. Easier to just recompile everything every time and make sure everything still works the way you expect.
So basically my approach is KISS, even if it means longer build times.