HNHacker News
TopNewBestAskShowJobs

lihaoyi

3,388 karma · joined December 23, 2011

submissionscomments
lihaoyi··on Ask HN: Those making $500/month on side projects in 2024 – Show and tell
Not quite 500$/month, but my book https://www.handsonscala.com/ is still making 300-400/month 4.5 years after releasing it. Not a lot of money compared to silicon valley FAANG salaries, especially given the amount of effort that went in, but it's a nice feeling to see the dollars trickling after so long
lihaoyi··on Bazel 8.0
I don't doubt your experience, I can only provide mine. I've seen one rollout from the inside (~1000 engineers), performed one rollout myself (100-1000 engineers), and talked to ~10 other companies trying to do their own rollouts (100-1000 engineers). Everyone has their own unique circumstances, and I can only speak for what I have seen myself (first or second hand)
lihaoyi··on Bazel 8.0
Empirically speaking, unless you are already a Bazel expert or are lucky enough to hire one, most orgs I've seen need at least 3-5 full-time engineers to become experts in Bazel and about ~1 person decade of engineering to roll it out (i.e. it takes those 3-5 guys 2-3 years). Adopting Bazel is not something you do over a weekend hackathon; the cost starts becoming reasonable once you have 100-200 engineers who can benefit from the 3-5 guys maintaining their build tool

For your small-ish project I'm guessing you have wants, but if the project grows at some point those will transition into needs and then you may have to make hard choices. That's my experience rolling it out Bazel myself and maintaining it for a growing organization and codebase over 7 years

I'm hoping the Mill build tool I'm working on can become an easier/friendlier alternative that provides many of the same benefits, even if it can't support the extreme Google scale that Bazel does. So if you think these things sound cool, but you don't want to spend 1 person decade on your build tool, you should check out Mill!

lihaoyi··on Bazel 8.0
Bazel does a lot of non-obvious things that other build tools don't, that become important in larger codebases (100-10,000 active developers):

- Automatic caching of everything. In most build tools, caching is opt in, so the core build/compile steps usually end up getting cached but everything else is not and ends up being wastefully recomputed all the time. In Bazel, caching is the default so everything is cached. Even tests are cached so if you run a test twice on the same code and inputs (transitively), the second time it is skipped

- Dependency based test selection: You can use bazel query to determine the possible targets and tests affected by a code change, allowing you to trivially set up CI to only run tests downstream of a PR diff and skip unrelated ones. Any large codebase that doesn't use Bazel ends up re-inventing this manually, e.g. consider this code in apache/spark that re-implements this in a Python script that wraps `mvn` or `sbt` (which are build tools that do not provide this functionality) https://github.com/apache/spark/blob/290b4b31bae2e02b648d2c5...

- Automatic sandboxing of your build steps in CGroup/NS containers, to ensure your build steps do not make use of un-declared files. In most build tools, this kind of mistake results in confusing nondeterministic parallelism and cache invalidation problems down the road, where e.g. your build step may rely on a file on disk but not realize it needs to re-compute when the file changes. In Bazel, these mis-configurations result in a deterministic error up front

- At my last job we extended these cgroups to limit CPU/Memory usage as well, which eliminates the noisy neighbour problem and ensures a build step or test gets the same compute footprint whether run alone during development or 96x parallel on a CI worker (https://github.com/bazelbuild/bazel/pull/21322). Otherwise it's common for tests to pass when run during manual development then timeout or OOM when run in CI under resource pressure due to other tests hogging the CPU or RAM

- Built in support for seamless shared caches: e.g. I compile something on my laptop, you download it to your laptop for usage. This also applies to tests: e.g. if TestFoo was run in CI on master, if i pull master and run all tests without changing the Foo code, TestFoo is skipped and uses the CI result

- Built in support for shared compute clusters: e.g. I compile stuff and it automatically happens in the cloud on 96 core machines, or i run a lot of tests (e.g. after a big refactor) on my laptop and it automatically gets farmed out to run 1024x parallel on the cluster (which despite running faster shouldn't cost any more than running 1x parallel, since it costs 1024x as much per-second but should finish in 1024x fewer seconds!)

- Support for deep integration between multiple languages: e.g. building a Go binary and Rust library which are both used in a Python executable, which is then tested using a Bash script and deployed as part of a Java backend server. Most build tools support one language well and others not at all, Bazel supports them all (not perfect but adequately)

If you never hit these needs, you don't need Bazel, and you probably shouldn't use it because it is ferociously complicated. But if you do hit these needs, most other build tools simply do not cut it at all.

We're trying to support some of these use cases and provide a simpler alternative in https://mill-build.org, but Bazel really is a high bar to reach in terms of features that support large monorepos

lihaoyi··on Starlark Programming Language
Starlark is definitely a mixed experience IMO, from my 7 years working with it in Bazel

On one hand, having a "hermetic" subset of Python is nice. You can be sure your Bazel starlark codebase isn't going to be making network calls or reading files or shelling out to subprocesses and all that. The fact that it is hermetic does help make things reproducible and deterministic, and enables paralleization and caching and other things. Everyone already knows Python syntax, and its certainly nicer than the templated-bash-in-templated-yaml files common elsewhere in the build tooling/infra space

On the other hand, a large Starlark codebase is a large Python codebase, and large Python codebases are imperative, untyped, and can get messy even without all the things mentioned above. Even though your Starlark is pure and deterministic, it still easily ends up a rats nest of sphagetti. Starlark goes the extra mile to be non-turing-complete, but that doesn't mean it's performant or easy to understand. And starlark's minimalism is also a curse as it lacks many features that help you manage large Python codebases such as PEP484 type annotations, which also means IDEs also cannot provide much help since they rely on types to understand the code

For https://mill-build.org we went the opposite route: not enforcing purity, but using a language with strong types and a strong functional bent to it. So far it's been working out OK, but it remains to be seen how well it scales to ever larger and more complex build setups

lihaoyi··on If not React, then what?
ScalaRx and Scalatags actually works a lot like React.js, in user-facing experience even if not in implementation. Unfortunately I was just a college kid and couldn't push it the same way the pros at Facebook could push react!
lihaoyi··on If not React, then what?
The basic idea is if you have n ways your page can display, with the old PHP/CGI whole page refresh model you needed to write n endpoints, but every interaction required a slow and disruptive whole page refresh

With pre-react single page apps, if you had O(n) ways your webpage could display, you needed to write code to account for n^2 ways that each of those n states could transition into another one of those n states

With react, you write code to render n pages, and get the fine grained interactivity for free with good (enough) performance. It basically removed a tradeoff/dilemma that had been facing web developers since time immemorial, so no surprise it was a big hit

lihaoyi··on If not React, then what?
We've been doing the O(n) thing forever with PHP and CGI since 1995, the issue is that it required constant whole-page reloads and was a poor experience that got worse the more interactive your webapps got. React let you do the same thing but made it fast(ish) with local updates

We actually see similar innovations on the FP side as well, with persistent data structures that allow structural sharing so your "copy and update" operations can be fast enough to be usable (even if still not nearly as fast as in-place mutation)

lihaoyi··on If not React, then what?
The fundamental magic of react is that it lets you write code that renders O(n) UI states in a functional manner (i.e. by returning the HTML you want) rather than O(n^2) UI state transitions in a mutable manner (by mutating the page via nested callbacks), and somehow makes it kind of fast-ish

People love complaining about React, but I don't know if they remember the pre-React world where you had to write the code to manage O(n^2) UI state transitions via recursive mutating callbacks. Simple things like "have a the list of photos shown always kept in sync with a label showning count of photos" was incredibly error prone, and the complexity of implementation went up quadratically with the complexity of your app (due to the n^2 state transitions you needed to worry about!)

Today React is not the only one to do this O(^2) to O(n) complexity reduction, but it was the first to go mainstream, and very much deserves its success in spite of its various shortcomings

lihaoyi··on How fast does Java compile?
All this is actually covered in the article

> These benchmarks were run ad-hoc on my laptop, an M1 10-core Macbook Pro, with OpenJDK Corretto 17.0.6. The numbers would differ on different Java versions, hardware, operating systems, and filesystems. Nevertheless, the overall trend is strong enough that you should be able to reproduce the results despite variations in the benchmarking environment.

> Again, to pick a somewhat easily-reproducible benchmark, we want a decently-sized module that’s relatively standalone within the project

> "typical" Java code should compile at ~100,000 lines/second on a single thread

> We explore the comparison between Gradle vs Mill or Maven vs Mill in more detail on their own dedicated pages

lihaoyi··on How fast does Java compile?
Notably, the benchmarks I provide are all single-core. You can get more if you modularize the project and spread out the load over multiple cores, and some of the other pages linked from the OP go into more detail
lihaoyi··on How fast does Java compile?
I don't think anyone checks their artifact signatures. I've changed signatures several times over the past few years (every time I get a new laptop I forget to copy over my keys) and haven't heard a peep from anyone using my open source libraries who actually noticed the change
lihaoyi··on How fast does Java compile?
Don't worry, I wrote this article about Java but my day-to-day work is Scala. We're lucky if we can crack 4,000 lines-per-second-per-core on my pretty vanilla Scala projects x_x
lihaoyi··on How fast does Java compile?
The benchmarks in the article do use the Gradle daemon, but not the Maven one, mostly because mvnd isn't the default.

For Gradle, if you turn off the gradle daemon, it gets even slower than the numbers presented going from 4+ seconds to 10+ seconds per compile:

    lihaoyi mockito$ git diff
    diff --git a/gradle.properties b/gradle.properties
    index 377b887db..3336085e7 100644
    --- a/gradle.properties
    +++ b/gradle.properties
    @@ -1,4 +1,4 @@
    -org.gradle.daemon=true
    +org.gradle.daemon=false
     org.gradle.parallel=true
     org.gradle.caching=true
     org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8 \

    lihaoyi mockito$ ./gradlew clean; time ./gradlew :classes --no-build-cache
    10.446
    10.230
    10.268
lihaoyi··on How fast does Java compile?
Author here, hope people find this article interesting!
lihaoyi··on The Pythonic Emptiness
I went and dug up the original code that caused an issue. Here it is:

https://github.com/python/cpython/blob/v2.7.1/Lib/cgi.py#L60...

Python's std lib `cgi.FieldStorage` object was falsy if it did not define any headers, even if it contained file data.

Thus my conditional trying to check whether a file was being uploaded "if request.field_storage" was going through the False branch when files were being uploaded but only in certain header-free scenarios not covered by automated and manual testing. This resulted in us dropping user data on the floor and losing uploaded files for a very large number of users before we realized and shut it off

The other sibling post contains another example where people may be confused, and Google pulls up others. But this is the concrete case that caused us to send out a hundred thousand apology emails to affected customers after losing their files

lihaoyi··on The Pythonic Emptiness
Python's truthiness behavior was the trigger for one of my worst ever bugs early in my career, which not only pulled in senior engineering but also marketing/comms and legal to help sort out the mess. Not a fan!
lihaoyi··on What's New in F# 9
I persoanlly found the syntax weird and the IDE support poor when i was playing with it in 2013

Both of these are surprising to me. It is an indentation language, similar to Python which I love. And it was by Microsoft, which makes Visual Studio which I also loved

Yet I found the syntax a lot odder than Python: multiple ways to call methods (C# style and Ocaml style), a wide selection of strange brackets, `.[i]` for array lookup instead of `[i]` (i think they fixed this eventually). And despite Visual Studio being great for C#, its support for F# wasnt up to scratch

I ended up settling on Scala, which is semantically very similar, but with an easier syntax (not perfect, but better) and decent Intellij suppory (again, nowhere near perfext!)

lihaoyi··on Mill Build Tool Issue Bounties, ~26kUSD up for grabs, ~14kUSD paid out
They are bounties, paid on completion at my discretion. Anyone worldwide who can receive international wire transfers can receive payment for them. I've paid out to people in quite a variety of places, from US to Europe to Japan to India to Africa
lihaoyi··on Mill: A fast JVM build tool for Java and Scala
I called it a build tool for Java and Scala because that's what its good at right now. The software industry is hugely varied, so I can't target everything at once. In particular, this tool started off targeting 100% Scala, but branches out to Java since they share a lot of concepts (classfiles, jars, assemblies, maven central, etc.)

But you are right that it is not JVM specific! In the docs there is an example of adding Typescript module support, and an initial strawman implementation takes about 100 lines of code. I'm hoping others can extend Mill to places where I do not have the time and expertise

I also opened up a 500USD bounty to add a strawman Python example, so if anyone wants to try their hand at writing the 100 or so lines necessary, here's the link :) https://github.com/com-lihaoyi/mill/issues/3862

lihaoyi··on Mill: A fast JVM build tool for Java and Scala
It should! I haven't had a chance to create an example myself yet, but there's a 500USD bounty open if anyone wants to take a crack at it https://github.com/com-lihaoyi/mill/issues/3549
lihaoyi··on Mill: A fast JVM build tool for Java and Scala
Maven is actually pretty behind in terms of JVM dependency resolution. Mill uses Coursier, same as my last company did, and when my last company switched from Maven to Coursier we saw a 2 order of magnitude speedup, with resolution commands that used to take 30min finish in a few seconds to give the exact same artifacts and versions.

I actually have no idea why these other resolvers are so slow, or why Coursier is so fast, but this slowness is very much a "maven" or "gradle" thing rather than a "jvm" thing. And Mill using coursier does significantly better!

lihaoyi··on Mill: A fast JVM build tool for Java and Scala
Mill uses the same daemon design as Gradle and mvnd. You do hit edge cases occasionally, but overall it works great
lihaoyi··on Mill: A fast JVM build tool for Java and Scala
You can, but why should you need to? Why can't the build tool take the plugin code directly off of disk, build it, and use it? This kind of orchestration of manual steps is what build tools are meant to be good at
lihaoyi··on Mill: A fast JVM build tool for Java and Scala
Starlark is great, but so is Scala. People underestimate how big the ecosystem is even for a niche language like Scala:

- Global publishing and distribution infrastructure

- IDE support in multiple IDEs

- A huge ecosystem of third party packages, both Scala and Java

- An excellent Scala standard library and Java standard library

- Good performance.

- Tooling! Jprofiler is great. Others use Yourkit or JFR.

- Mill leans havily on Scala's FP/OO hybrid style with types, while starlark provides none of that and is purely untyped procedural code

lihaoyi··on Mill: A fast JVM build tool for Java and Scala
The test thing is just a matter of defaults, in Mill subprocess testing is the default and in-process testing opt-in via .testLocal.

I also believed that a lot of existing JVM tooling is bloated and slow, so we are in agreement! Mill tries to be different, so do give it a chance if you can. There is life beyond Maven, Gradle, and SBT

lihaoyi··on Mill: A fast JVM build tool for Java and Scala
The intention has changed, Mill now explicity targets Java and Kotlin as well. It now has dedicated Java/Kotlin docsite sections and examples, and has grown integrations with Palantir-Format, Checkstyle, Errorprone, Jacoco, and all their Kotlin equivalents (ktfmt, ktlint, kover).

Java and Scala (and Kotlin) are remarkably similar from a tooling perspective, so Mill tries to target both using the same shared infrastructure

lihaoyi··on Mill: A fast JVM build tool for Java and Scala
Dependency resolution uses Coursier, which is one of the open source JVM dependency resolvers. SBT uses it too, and my last company used it with Bazel

The "ivy" thing is legacy haha. Mill used to use Apache Ivy to resolve dependencies, years ago. Coursier was a better/faster replacement, but names have a tendency to stick around

lihaoyi··on Mill: A fast JVM build tool for Java and Scala
Depending on when you tried, it could be worth trying again: support for Java has improved greatly in the last few months, as have the documentation. Come by our discord channel if you get stuck and i can help unblock you
lihaoyi··on Mill: A fast JVM build tool for Java and Scala
Even in Java, because the language is relatively verbose, many frameworks fall back to an "inner platform" of magic annotations which have the same problem: e.g. just because you know how Java works doesnt mean your mind hasn't unloaded all the SpringBoot annotation semantics! But despite that it is worth it, because conciseness does matter.

Mill using Scala syntax is like that, but with the added advantage that even if you forget how Scala works, your IDE does not. You can really lean on Intellij or VScode to help you understand and navigate around a Mill build in a way that is beyond what is possible for most build tools: You can autocomplete things, peek at docs, navigate the build graph and module tree, etc. and learn what you need to learn without needing to reach for Google/ChatGPT. I use this ability heavily, and I hope others will enjoy these benefits as well

← PreviousPage 2 of 9Next →