Yes, it's complicated, but it's also quite rigorous, and the rigor pays off.
(We at Square had already found a Blaze-alike necessary. We are currently busy converting our Java build from Pants to Bazel.)
Yes, it's complicated, but it's also quite rigorous, and the rigor pays off.
(We at Square had already found a Blaze-alike necessary. We are currently busy converting our Java build from Pants to Bazel.)
Both approaches take a huge amount of work and tooling.
The big selling point of a monorepo is that the time and effort taken to follow strict versioning and upgrade discipline for multiple interdependent projects can be somewhat avoided. On the code side.
Essentially: version skew across numerous artifacts in a large organization starts to look like the version skew across an industry or ecosystem. The aggregate cost of dealing with it project by project is probably higher, at least that is what most of the biggest tech companies have concluded, than dealing with it at the source level using a monorepo and single-version policy.
EDIT: Looks like I've misread the parent's argument as one against monorepo. It was in fact an argument in favor, and one I agree with.
Everything is hard, unfortunately.
Doing that in one repo makes the commit part easier, but hides the complexity of deploying separately. Or to other places.
Not that two repos makes it easy. Just gives a much earlier signal to where it happens.
In my experience, you're required to break changes up into safe individual deployments anyways, so the monorepo doesn't add any benefit in that sense.
Having looked seriously at both options, I think the monorepo world is the right one, but it presently lacks good tooling to sanely model your dependency graph AND create custom build rules while still being affordable for small or medium-sized orgs. Git/hub simply isn’t designed for this kind of modeling and everything I’ve seen built atop it is either way too manual or a kludge. Maybe the “kludge” solutions are actually reasonable, but my confidence is low.
Bazel is the right idea, but it’s execution disappoints. The documentation is abysmal, last I checked they advertised Python 3 support, but it’s been broken for years with no signs of progress. Building custom rules also looked hopelessly complex (by which I mean, “not something our organization can afford to implement and maintain”) but maybe there’s some undocumented happy path that I’m missing out on? These things seem easy enough to implement. We’re using Pants right now, and for it’s many similar problems (bugs, documentation, poor code base, difficult extensibility), it at least does a passable job at building Python projects.
I’ve thought about it a fair amount, and I think it’s reasonable to build something simpler that might not meet Google’s use case, but would at least enable small and medium sized shops to play the monorepo game.
The next obvious question is, what would you do to make it simpler? Tons of people have tried (you listed 5), and they all rebuilt the same thing. What features do you drop?
I understand that "it should be simpler" is a pretty lazy criticism. It's been a while since I audited Bazel and friends, and I've forgotten which issues apply to which tool. Moreover, because of the awful state of the documentation and the messiness of the code base (or perhaps this is just standard quality for Java projects?), it's really difficult to tell whether any given issue is actually a fundamental shortcoming in the application or whether it's simply a knowledge gap.
As far as what I want, keep the starlark configuration file format; implement all rules as starlark libraries (such that no one needs to write Java to extend, and if you must write Java then for goodness' sake fix the plugin interface or document it better or something such that one doesn't need to be a core contributor to implement a plugin--perhaps this is fine for an enterprise audience, but it's not fine for my use case). The rules should call into a base `mktarget()` or similar that takes args like the target's ID (the package:target_name pair), a target type that identifies the code used to build the target, and a dict of args/params that are passed into the aforementioned code. The args/params can be an arbitrarily nested JSON-like type so long as the leaves are primitives (int, string, etc), references to source files, or other targets and all leaves (and transitively, the whole structure) must be hashable such that we can identify a given execution of the build.
Beyond that core operating model, the code and the user interface should be clean and well documented. Ideally, small and medium-sized projects shouldn't need to run it in daemon mode to get reasonable performance. This is important because a daemon running on local development machines introduces a larger maintenance burden (there's just more that can go wrong). Language-specific plugins (custom rules, whatever you want to call them) should adhere pretty closely to the conventions of the target language. Lastly, there should be good support for building toplevel artifacts--this means I should be able to build a whole CloudFormation package including lambdas, Docker images, etc just like I would build a JAR or a C++ binary.
I realize that those things are easy enough to say, but the devil is in the details. I've actually gone so far as to prototype the implementation, so I'm confident that those goals are achievable. Unfortunately, it's a pretty significant effort (mostly due to the breadth of project types/languages to support and the nuance/expertise required to support any of them), so I'm bound by free time. If anyone is interested in collaborating or discussing more in-depth, hit me up on Twitter @weberc2 or email me (username at gmail.com).
we don't use bazel (yet), because dotnet is not that supported.
You will soon experience:
- dependencies hell due to transitive and conflicting dependencies
- one back-incompatible change in some obscure library end up breaking some other unknown service that happens to transitively depend and it
- the entire codebase will become a mess due to inconsistent code styles and formatting because hey we are developers and we can never agree on anything. Thus each team lead will have its own opinion
- each team will have to maintain its own CI/CD jobs
- heterogeneous builds: maven, node, sbt, webpack, etc ...
the list goes on ...
All (or most of) this mess is solved by centralizing the codebase in a monorepo.
Bazel goes through great pains to only rebuild the minimum necessary for correctness. It's able to do that because bazel build files get a lot more information about the source level dependencies than a Nix file does