3,388 karma · joined December 23, 2011
You don't need to globally install Apache-Commons, nor is there a "Apache Commons Version Manager" that you need to first install to then manage your different versions of Apache Commons. So why should the JVM need such a thing? Well with Mill, it doesn't
- https://github.com/orgs/com-lihaoyi/discussions/6
If you look at that thread, you'll see I've paid out quite a lot in bounties, somewhere around 50-60kUSD (the amount is not quite precise, because some bounties I completed myself without paying, and others I paid extra when the work turned out to be more than expected). In exchange, I did manage to get quite a lot of work done for that cost
You do get some trash, it does take significant work to review, and not everything is amenable to bounties. But for projects that already have interested users and potential collaborators, sometimes 500-1000USD in cash is enough motivation for someone to go from curious to engaged. And if I can pay someone 500-1000USD to save me a week of work (and associated context switching) it can definitely be worth the cost.
The bounties are certainly not a living wage for people, especially compared to my peers making 1mUSD/yr at some big tech FAANG. It's just a token of appreciation that somehow feels qualitatively different from the money that comes in your twice-monthly paycheck
This all comes built-in without any plugins or anything, and serves to help speed up dev and CI workflows even on pretty large codebases.
Mill is intended to be much easier than Bazel. Most people would not use Bazel for a 1-person project, or even a 5-person project, and it only starts pencilling out once you have 50-100 engineers on the team. Mill in contrast works great for small projects and medium projects as well.
One way I'd look at it is that Mill works great for 1-500 person projects, while Bazel works great for 100-5000 person projects. There's some overlap in the middle, but fundamentally they target different kinds of users who have very different constraints and requirements
That's really the case with Mill as well, at a deeper more-meaningful level. Mill builds are all built around objects, classes, methods, and overrides.
Maven is XML. Do you write large codebases in XML? Gradle is Groovy/Kotlin, but it's an odd Groovy/Kotlin dialect or DSL that looks totally unlike application codebases, and your IDE is unable to understand and navigate effectively.
Mill is Scala, sure, but what's important is that it is architected around objects, classes, methods, and overrides. Those are the core abstractions when working with Mill, which are the same core abstractions as any Java/Scala/Kotlin application. Thus IDEs are able to navigate Mill builds, profilers like JProfiler or Yourkit can work on Mill builds, and humans who learned Java 101 in college are immediately familiar with the structure and fundamental abstractions of Mill.
For example, in Maven you typically extend your build in Bash scripts + maven-exec-plugin, Ant script + maven-antrun-plugin, or custom Maven plugins entirely. None of these are "nice" programming environments, with proper IDE support, typechecking, introspectability, and so on. Writing lots of logic in Bash or Ant is risky, so you would be right to minimize writing code in it
Similarly, in Gradle you extend your build in Groovy/Kotlin, but it's a kind of "weird" Groovy/Kotlin unlike anything you'd write in application code. For example, your IDE support in Gradle-Kotlin is much worse than what you get in normal-Kotlin. Despite Gradle-Kotlin being the same language and same IDE, it's a much worse experience writing it, it's much easier to make mistakes, and so you are right to minimize the code you write in it.
In Mill, the build scripts are in Scala, but that's not the important part. The important part is you write normal code using classes, methods, and overrides. IDEs are very good at navigating classes, methods and overrides, and developers are very familiar with working with classes, methods, and overrides. And so build code in Mill feels as comfortable as your application code in Java/Scala/Kotlin: same quality of IDE experience, same typechecking, even can use the same Java third-party libraries (if you wish).
So it's understandable you hate programming your build. In Maven or Gradle or SBT, I'd hate programming my build as well. What Mill offers is that you can program your build where necessary without the hate that comes with doing so in other build tools!
Mill generally is able to replace a lot of first-party/third-party extensions with builtin functionality: beyond the plugins above, it also subsumes autoformatting plugins, linting plugins, external tools like sdkman/jenv, and so on. Thus when using Mill you often don't even think about "what plugins I need", because the bulk of common use cases are provided out of the box, and you can instead focus on your actual project and application code
- https://mill-build.org/blog/2-monorepo-build-tool.html
People like to rave about Monorepos, and they are great if set up correctly, but there's a lot of intricacies that often goes on behind the scenes to make a Monorepo successful that it's easy to overlook since usually some "other" team (devops teams, devtools team, etc.) is shouldering all that burden. Still worth it, but most be approached with caution
Build tools are generally an un-sexy field, and JVM build tools perhaps doubly so. But Mill demonstrates that with some thought put into the design and architecture, we can speed up JVM development workflows by 3-6x over traditional JVM tools like Maven or Gradle, and make it subjectively much easier to navigate in IDEs and extend with custom logic.
If you're passionate about developer experience and work on the JVM, I encourage you to give Mill a try!
IMO Bazel has a lot of good ideas to it: hierarchical graph-based builds, pure hermetic build steps, and so on. Especially at the time, these were novel ideas. But in Bazel they are buried behind a sea of other concepts that may not be so critical: `query` vs `aquery` vs `cquery`, action-graph vs configured-action-graph vs target-graph, providers vs outputs, macro vs rule-impl, etc. Some of these are necessary for ultra-large-scale builds, some are compromises due to legacy, but for the vast majority of non-Google-scale companies there may be a better way.
I'm hoping the next generation of build tools can simplify things enough that you don't need a person-decade of engineering work to adopt it. My own OSS project Mill (https://mill-build.org/) is one attempt in that direction, by re-using ideas from functional and object-oriented programming that people are already familiar with to make build graphs easier to describe and work with. Maybe a new tool won't be able to support million-file monorepos with 10,000 active contributors, but I think for the vast majority of developers and teams that's not a problem at all.
My experience was that it was a surprisingly nice language with a surprisingly warty user experience: papercuts ranging from naming conventions and function call styles (`|> List.map` vs `.Select`), basic syntax (`foo.[0]` to lookup arrays), type system features (F# doesn't have covariance/contravariance even though C# does), IDE support (back then was only Visual Studio, whose support for F# was inferior to C#).
Ended up settling on Scala after that, as a language with its own Warts, but one that somehow managed to feel a more cohesive than F# did despite having largely the same featureset and positioning.
F# was my first functional language and one that changed how I look at programming, but at the same time I'm happy to not actually have to use it for serious programming!
IIRC Haxl (https://github.com/facebook/Haxl) uses Applicatives to optimize and parallelise remote data fetching, which is hard to do with Monads since those are inherently sequential due to the nature of `flatMap`/`bind`. My own Mill build tool (https://mill-build.org/) uses an applicative structure for your build so we can materialize the entire build graph up front and choose how to parallelize it, query it, or otherwise manipulate it, which is again impossible with Monads since the structure of a Monad computation is only assembled "on the fly" as the individual steps are being evaluated. "Parallel Validation" where you want to aggregate all failures, rather than stopping at the first one, is another common use case (e.g. https://hackage.haskell.org/package/validation or https://typelevel.org/cats/datatypes/validated.html)
Monads seem to have this strange aura around them that attracts certain kinds of personalities, but really they're just one abstraction in a whole toolkit of useful abstractions, and there are many cases where Applicative or some other construct are much more suited
- You usually have shared initialization nearer the root and the various cases you want to assert at the leaves.
- You want to group related tests logically together, so it's not one huge flat namespace which gets messy
- You want to run groups of tests at the same time, e.g. when testing a related feature
Typically, these different ways of grouping tests all end up with the same grouping, so it makes a lot of sense to have your tests form a tree rather than a flat list of @Test methods or whatever
Naturally you can always emulate this yourself. e.g. Having helper setup methods that call each other and form a hierarchy, or having a tagging discipline that forms a hierarchy to let you call tests that are related, or simply using files as the leaf-level of the larger filesystem tree to organize your tests. All that works, but it is nice to be able to simplify define a tree of tests in a single file and have all that taken care of for you
I've been working on an alternate build tool Mill (https://www.mill-build.org) tries to provide the 90% of Bazel that people need at 10% the complexity cost. From a greenfield perspective a lot of work to try and catch up to Bazel's cross-language support and community. I think we can eventually get there, but it will be a long slog
(define (match:element variable restrictions)
(define (ok? datum)
(every
(lambda (restriction)
(restriction datum)
)
restrictions
)
)
(define (element-match data dictionary succeed)
(and
(ok? data)
(let ((vcell (match:lookup variable dictionary)))
(if vcell
(and
(equal? (match:value vcell) data)
(succeed dictionary)
)
(succeed (match:bind variable data dictionary))
)
)
)
)
element-match
)
A lot of "staged" programs can be thought of as semi dynamic as well, even things like C++ template expansion or Zig comptime: run some logic up front, freeze it, then run the rest of the application later
50USD gets you about 960 linux core-hours on AWS. A lot more than 90 minutes on 1 box on Windows/GHA, but not so much that someone running a lot of unit/integration/e2e tests in a large codebase won't be able to use
There's a lot I can respond to in this post, but I think the bottom line is: if you have not experienced the problem, count your blessings.
Lots of places do face these problems, with test suites that take hours or days to run if not parallelized. And while parallelization reduces latency, it does not reduce costs, and test suites taking 10, 20, or 50USD every time you update a pull request are not uncommon either
If you never hit these scenarios, just know that many others are not so lucky
Let's say you have 100 small repos, and make a change to one. How confident are you in the changed repo's test suite that you can guarantee there are no bugs that will affect other related repos? If not, which other repos do you test with your change to get that confidence?
> Note that while Mill's first target is Scala builds (as an alternative to SBT) there isn't much reason why Mill shouldn't be usable for other JVM/non-JVM languages too.
Now 7 years later, Mill indeed has first-class support for Java (https://mill-build.org/mill/javalib/intro.html) and Kotlin-JVM (https://mill-build.org/mill/kotlinlib/intro.html). There's experimental work ongoing to expand that support to Kotlin-Multiplatform, Android, Python, and Javascript/Typescript, with ~20,000USD of bounties up for grabs (and a similar amount already paid out):
- https://github.com/orgs/com-lihaoyi/discussions/6
We also recently started an engineering blog on build-engineering topics
- https://mill-build.org/blog/
Take a look at the open bounties and get involved if you are interested in this kind of fast, easy, multi-language build tooling!