Hacker News Reader

Comment by swiftcoder | Hacker News Reader

This is where letting a large enterprise guide the development of a piece of widely-used software becomes questionable. At a FAANG the constraints are fundamentally different.

At work I routinely see CLIs programs clocking in at a gigabyte, because it was simpler to statically link the entire shared dependency tree than to figure out the actual set of dependencies, and once your binary grows that big, running LTO adds too much time to one's builds. And disk space is basically free at FAANG scale...

View on HN

> This is where letting a large enterprise guide the development of a piece of widely-used software becomes questionable. At a FAANG the constraints are fundamentally different.

I see where you're going with this, but the conclusion of the article is that Go isn't (what you call) FAANG-y enough for them.

From the article: "This design choice was intended to lower the start-up time of programs [...] This performance goal is not relevant to server software with long-running processes, like CockroachDB, and its incurred space cost is particularly inconvenient for large, feature-rich programs."

What they want is a way to pass 1000 parameters to the compiler so that nobody ever gets a working binary unless they have a team of release engineers to make one for you. The Go team and their users at Google easily have access to said team of release engineers, but took the opposite approach. They optimized for programs that start fast even though most of those teams at Google are writing programs that run for a long time. They optimized for a compiler and runtime that don't have a billion knobs to turn, even though they can afford to pay an army of engineers to tune those knobs.

So I actually think this has very little to do with optimizing for FAANG. This is more about every program getting the same compiler options, which is merely a philosophy that some people on the Go team have, not really a philosophy that every FAANG has. (They use Java at Google too, and Java is more than happy to give you knobs to turn. It also doesn't start up very fast.)

> It also doesn't start up very fast

This has been changing rapidly with GraalVM and projects like Quarkus, which can build a static binary of a Java program that starts, as you'd expect, much faster (still not performance of a C/C++ program, but way faster than a regular Java program).

Did I miss something important about GraalVM? I thought it was another runtime on top of JVM, not a replacement for it. For a Java program, for example, GraalVM would presumably have no impact since it wouldn't be used for it.

GraalVM is several things.

In this context people are talking about the native-image tool which uses a small JVM written in Java (SubstrateVM) to compile to native code ahead of time, no HotSpot or other JVM dependencies.

There's also Truffle which is a way to put other languages onto the JVM (both HotSpot and SubstrateVM).

Java starts up pretty fast if you bother to AOT compile it, plenty of commercial options available since early 2000's.

For the free beer folks there is AOT support on OpenJDK, OpenJ9 and GraalVM.

Maybe with bad C toolchains you can get combinations of switches that result in dead binaries, but I've never seen a JVM that wouldn't start because of bad tuning flags. At most you can hurt (or help) performance.

I'm not sure the Go 'philosophy' here has anything to do with it. If you click through some links you'll end up at a GitHub issue where they discuss a prototype for making Go's code generation not completely awful. The justification for not pursuing a 5-10% improvement (which is huge for compiler opts) is that it'd break existing inline assembly by changing the calling convention. This isn't a philosophical objection, it's just bad design painting them into a corner.

Disk space in general is pretty much free these days. 123mb for a whole database is really not that big of a deal, IMHO. For example, my local PostgreSQL docker image is 140mb plus the alpine OS (5mb). And the Ruby on Rails application using that image clocks in at a little over 1gb, also using the ruby alpine image as a base (50mb).

With my company, the cost really started to become a burden with data transfer. But transferring images to and from the AWS container registry is so expensive that we actually build production images inside the Kubernetes cluster (plus the cluster has access to all the secrets and stuff), even though it was a bit harder to implement.

If you're FAANG and you can run your own stable and highly-available cloud, data transfer rates don't matter, so you can deploy your application the "right" way in a containerized world.

Yes but when running a long-lived application on a lot of data it's typically important to keep the executable small both so it can be "hot" and to leave more room for data. At scale this could be even more important, not less, than for a smaller operation.

Of course the real (i.e. explicitly stated by Pike) driver for go was the assertion that inexperienced new hires write poor code and so the harder it is for them to get into trouble the better, even at the cost of other issues.

Executable size on disk doesn't dictate effective size at higher levels of the memory hierarchy.

IOW, you're paying by the page/cache line. If the extra bloat (debug information) in your executable isn't interspersed within your actual code (it shouldn't be), you aren't paying for it in runtime efficiency.

> Yes but when running a long-lived application on a lot of data it's typically important to keep the executable small both so it can be "hot" and to leave more room for data.

Who really runs server applications these days where data/rss are not a large multiple of the actual code segments? People happily run JVM server processes these days, how much code/data do you think that pulls in just to start up?

Genuinely wondering how a CLI would be 1GB, even at Facebook. Never encountered anything _close_ at Amazon.

I never saw one a singular tool that big at Amazon either, but the general concept held. The AWS service I worked on had ~1,000 jars in its dependency closure. All of those had to be uploaded to S3 after each build, and then downloaded onto each EC2 instance during deployment.

We're talking on the order of a terabyte of data transfer each time we deployed to a thousand instance fleet (ideally deploying weekly)

Not your usually usecase, but the binary for running the Bloomberg terminal is well over 1GB (In fact running into the the 4GB executable boundary was a problem).

Oh, that's fascinating. I never checked the binary size but that makes sense.

I’m curious, what’s in it? Which sections take up the space?

Sorry, I should have clarified, this is the server side binary. I believe the part that's run on the user's computer is just a kind of dumb terminal.

The reason it's so big (or at least was several years ago) is that every single function is compiled into one binary monolith.

I’m sorry, that doesn’t sound possible. A clean Windows 10 install (kernel plus userland) can clock in at 6GB and you’re telling me the Bloomberg server CLI was hitting the 4GB wall just because it’s statically linked? No amount of pure code gets that big, it must be something else in the tooling or object format. Perhaps it includes binary assets.

That's quite believable to me for an old codebase. Command line utilities at Google routinely reached hundreds of megabytes in binary size due to static linking of everything. A Bloomberg server is surely far larger and more complex than those tools, so gigabytes isn't unbelievable.

Binary code can be quite bloated compared to source code. It depends a lot on the encoding. The same author as this blog post has another actually much more interesting one on how Go compiles common constructs. It's rather enlightening as the example generated assembly (implicitly) explains why Go compiles so fast. It has to be the dumbest, least optimising compiler I've ever seen. Very short and trivial source code constructs like function calls balloon into pages of instructions, far far more than a C compiler would generate.

C++ compilers also have a reputation for generating bloaty code, but that's mostly due to compile-time specialisation of containers and the way template expansion works. If you write C-as-C++ then you get pretty tight binaries.

I think you're underestimating just how much the bloomberg terminal backend does. It's 40 years of obscure functions packed in to one massive binary.

Mmmm I don't know if I agree with that logic.

I'd argue the opposite -- as a startup, you can't afford to micro-optimize. Labor, time and opportunity cost dwarf all but the grossest resource waste. If you need to use 100GB/k8s node instead of 50GB, it will have 0 effect on the success of your venture.

At Google scale, it becomes worth it to optimize:

- You are delivering more product per engineer, and more product means more resources. Instead of a single customer instance which costs $1/month more, you have 100,000 customer instances, costing a significant amount more. It becomes worth trimming margins.

- You have economies of scale, and it might be worth it for an engineer to spend a month trimming 2% of the cost of a software deliverable.

The common refrain for startups is "do things that don't scale", and this is for good reason. Google has to actually worry about fixing things AFTER they are scaled.

I think you're both right.

A 125MB binary is relatively beefy, but still easily fits in RAM. The amount of disk that you're spending on a single executable (your database, in this instance) is tiny in comparison to the amount of data stored in that database.

It's definitely worth it for Google to trim 2% off of their storage requirements - but if your binary is 0.1% of your storage, it's barely even worth glancing at.

There are plenty of efforts and tooling that reduce the number of dependencies and the deploy size of binaries at Google. The notion that we don't care about size isn't true.

But it's true that it's not worth optimising first, it's done by first evaluating the impact across the fleet and then prioritising the most effective changes.

LTO is no magic bullet for binary size either. A binary that does nothing will still link in the whole C library. It doesn't end up decreasing large program sizes that much in my experience either.

Tree shaking?

Reply on news.ycombinator.com