Alternatively don't use monorepos, and take advantage of binary libraries.
Naturally there is the caveat that using binary libraries isn't a thing in Go.
Naturally there is the caveat that using binary libraries isn't a thing in Go.
You either:
* Just don't test such changes at change time, break down and push the testing effort to downstream projects in disguise of "dependency management".
* Structure your engineering effort to avoid such changes, for example avoid having a base library shared between teams.
Neither is ideal for a corporation environment, because both harms velocity. We accept these in the "open source world" because we don't have better options, but the same does not hold for corps.
And now you get into the fun situation of having to handle ecosystemic asynchronous updates between your multiple repositories.
Our services aren't even particularly well-factored and we update median 2, mode 1, mean 3-4 when we change a core library. (Out of ~25.)
This isn't true in my experience at large companies. For this to be true, you would either need to have absolutely no shared core libraries, or core libraries that are so stable and unchanging that they never receive updates. Updates to those core libraries usually result in a cascade of changes across a ton of repos, tons of tests that need to be rerun, and a slew of now incompatible versions.
All of these problems go away with a monorepo. Monorepos are absolutely the right solution for 98% of companies out there because they will never reach the engineering scale where monorepos start to struggle. For the 2% of companies that do reach that engineering scale, these kinds of blog posts get written.
The mistake is reading blog posts like this and thinking is necessary or even relevant to any more than 2% of the companies out there. The vast majority of shops will never need anything like this.
I don't really see this as a mono vs. multi repo issue either. In a mono repo you can still choose which projects update and the more you choose the bigger the diff and more involved the review process. In a multi repo case you can build analogous tooling to generate downstream MRs automatically.
If you need to touch every downstream project regularly just to keep things functional, you don't have a mono-repo, you have a monolith. (Which can also be fine, really! But it's not the same thing at all.)
(Yes, I know with 25 projects we're nowhere near Uber scale. But based on my experience at larger companies, this seems to scale more or less linearly - in terms of downstream project count, library count, and developer time available to dedicate to such things. If anything, developer time available is what seems to scale super-linearly.)
Even a security fix is unlikely to affect every downstream user of a library except in egregious cases. But, if so, yes you update them all. (I'd say this is much less frequent than, I don't know, OpenSSL or log4j having a bug that makes us do this. The specific concern of an internal library having such a broad vulnerability is negligible in influencing our CI design.)
> What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once?
You do it.
> What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things?
You do it.
I didn't say "it has no associated costs." But you need to weigh those costs against the other costs of a monorepo, and other costs in your CI/CD design generally. "Take longer to update a really old project once a year" is a much lower cost for us than "have a dedicated CI team to wrangle the tooling we need for automatic downstream pushes / a monorepo."
It's unusual because a multi-repo environment makes it difficult to do, not because it's not necessary.
> We do that with all external dependencies.
Indeed. Why you’d want to also suffer that for internal dependencies when there is no reason to, I can’t fathom.
If you have a good process for external dependencies you can apply that to internal. If you don't have a good process for external dependencies a monorepo won't help (unless you make all your dependencies internal, which is even more expensive)
Yeah... but usually good process for external dependencies requires a lot of ceremony and takes maybe days to update one dependency. If the same happens to internal dependencies we will be just slower at making changes for no good reason.
This is more of a cultural thing. In fact, we are creating complicated solutions to mitigate this problem, only to go full circle after a while: people implemented Service Mesh via proxy sidecars, it added more layers (than just embedding this into the RPC library), but even doing so is easier than persuading downstream project owners update their dependency in a timely manner, so it takes off. And now they are going to chase for "proxyless" due to "performance". Oh well.
Also, having a rather global view of the impact of my code before actually make the change helps a lot, especially if you are working on low-ish level libraries. Technically this is not bound to monorepo, but people usually criticize the complexity introduced by having such ability for monorepo, so :)
> the extra time to make multiple PRs is dwarfed by the actual update's complexity
Writing code is never the majority of the work. All the operational stuff is more complex without forced upgrades: coordinating the dependency updates, automating integration testing, pushing clients to migrate off old thick clients, determining what version of a specific repo is deployed, running a bisect to root cause issues, etc.
At that scale, anything reducing complexity is a bonus. At that scale, monorepos are great for that.