Hacker News Reader

Comment by danso | Hacker News Reader

I don't work in a shop where performance/speed is important, but I am looking for other ways to do things I would do in Make but...not in Make.

For example, my use-case is similar to what Mike Bostock described in "Why Use Make" [0] when explaining how he uses Make to build out his data tranformation process. Most of my work is data transformation/small-scale ETL, but I just haven't been able to get into Make beyond trivial work, and I often end up writing things in Rake (Ruby).

So I was wondering if other devs had tried using Buck/Bazel for everyday hobbies and projects, and whether you stuck with the new tool or went back to Make? The portability of Makefiles isn't a high priority for me, and I like experimenting with different systems for my own projects.

[0] https://bost.ocks.org/mike/make/

View on HN

I tried Buck and found it to be a poor clone of the Google build tool it's based on. If you're interested, go straight to Bazel -- it's the real thing.

Edit to add: here's a specific complaint. To run arbitrary commands and shell scripts, you use genrule(), but in Buck a genrule can only have a single output. I used Buck to preprocess and organize the assets for a game, and that restriction made it very awkward.

It's been a while since I used Buck, and it looks like this has improved a bit -- you can now output a folder of files: https://buckbuild.com/rule/genrule.html

Here's the top-level Bazel BUILD file of the open source project I work on at Google, if anyone wants to see an example for a good-sized (~100K LoC) codebase: https://github.com/google/nomulus/blob/master/java/google/re...

Each subdirectory below the root also has its own BUILD file, where you can see the individual packages being compiled.

These build systems become super handy once you're working on a large tree of many different software projects. For small projects not so much. It's probably easier to use the default build system of your programming language.

When make fails me for personal stuff, I switch to ninja [1]. The Chromium people use it in their Bazel-like build system. A hidden gem I recently discovered is doit [2]. I found this one incredibly helpful when I had a build with tons of 1:N, N:1, N:M dependencies.

[1] https://github.com/ninja-build/ninja [2] http://pydoit.org/

Thanks for the suggestions! That "doit" looks like it's simple enough for my fairly pedestrian needs. I wonder if my aversion to Make is that I don't spend enough time thinking/making Makefiles, but having something a little friendlier and simpler -- at the cost of ubiquity/portability -- might be the kind of training wheels I need.

Edit: FWIW, I posted doit to r/python and someone suggestion Snakemake, which also looks excellent and well-maintained: http://snakemake.readthedocs.io/en/stable/

> A hidden gem I recently discovered is doit [2]. I found this one incredibly helpful when I had a build with tons of 1:N, N:1, N:M dependencies.

> [2] http://pydoit.org/

Thanks for this! I need a build tool for a Python based data analysis project ("micro ETL"?). Doit looks like it will work nicely, with the bonus of keeping everything in Python.

So far I've also always found `make` to be the best option for what I need, especially because it's already available everywhere (albeit sadly in very subtly incompatible versions).

Anyways, one thing that has always bothered me about make is that it so much depends on file modification dates. Imagine if it would instead use a very optimised hashing algorithm over the input content. Content can be a file or any URI, so it uses wget/curl and ssh as a dependency. Hashing is optimized such that it fails early - e.g. hash in n kB increments, mark as new and return if changed.

Imagine how well this would now suddenly integrate into our modern web service based landscape. You could hook together services quite easily:

    all: report.csv

    report.csv: ${customers_json_uri} ${earnings_json_uri}
        ${report_executable} $^ > $@

Has anyone built something like this already?

Buck (and Bazel) essentially do what you want and more:

- Don't use modification times and instead do intelligent hashing (see https://buckbuild.com/concept/rule_keys.html)

- Include the compiler and a bunch of other stuff in the hash. This means you can share cached artifacts safely via a http cache (https://buckbuild.com/concept/http_cache_api.html)

- Can refer to remote files (see https://buckbuild.com/rule/remote_file.html)

- Can use any script to do anything as long as it has a single consistent output (see https://buckbuild.com/rule/genrule.html)

Because their model is so much better, you get crazy fast local caching and remote shared caching of artifacts.

Please don't use make, the newer build tools are better in virtually every way.

(disclosure: my team created Buck when I was at FB)

I have briefly used https://github.com/Factual/drake for this.

It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover:

* Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways,

* Outputs that are directories with many files,

* Built-in input/output file sharding (for example by date or month) and HDFS support,

* Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc,

* Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.

I'm not commenting about Drake or its usefulness, but concerning (GNU) make :

* you can specify several inputs & outputs per rule

* you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell...

* downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided

* fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps)

and

* "everyone" "hates" gnu make, which means everyone knows it.

(edit :format)

AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.

Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multi...

That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.

That doesn't really look like "handling it" by any sensible definition. That looks like a huge kludgy workaround, almost like user implementing a build system inside the build system. (I mean, wtf, asking the user to implement a lock-and-set operation inside their build?)

GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.

It even has builtin implicit rules. Enabled by default. Checking for the existence of files that are not found in any new project for maybe ten or twenty years (such as RCS and SCSS version control files).

On. Every. Single. Run.

Yup.

But I love the built-in implicit rules. I use these frequently.

Often, I write a little test program in C or C++. I name the file "foo.c". Then, to build it I simply type `make foo`.

Sure the rules may be a bit stale, but there isn't much software i use today that I could use in a near identical manner 40 years ago. Make has truly stood the test of time.

Closest in spirit to make, and does essentially all you want to do is "redo", concept drafter by djb, implemented independently by apenwarr[0] and by others. apenwarr's version has a super short "do" script (a couple of hundred lines of portable sh IIRC) which uses the same configuration but does a full rebuild each time.

The two weak points are: (1) multiple outputs from a compilation step (yytab.c yytab.h from yacc) is not properly supported, and (2) no windows support. Other than that it's the perfect minimalistic make replacement -- what make should have been.

Additionally, there tup[1]. With some assumptions about the build process that usually hold in non-distributed builds, it is the fastest, simplest make replacement; You just write a list of commands that builds your final outputs, and by tracing the processes it figures out exactly what needs to be done next time -- nothing more, and nothing less.

[0] https://github.com/apenwarr/redo [1] http://gittup.org/tup/

Buck (and Bazel) both have rules to run and cache arbitrary rules/scripts as part of the build, assuming they produce one output:

https://buckbuild.com/rule/genrule.html

Some other benefits that may or may not be important:

- Use the same build system for iOS/Android/Cxx/Python/D, etc.

- Integrates with editing in IDEs.

- Encourages modularization...more modules = more cache hits = faster builds

I have used Bazel (on Mac OS) to build a small C++ project. The nice thing about it is that it can be easily configured to download and build third-party dependencies from github. I have used this feature to add gflags, protobuf, and googletest dependencies to my project. That being said, it is still not perfect. For instance, I was not able to build GzipInputStream and GzipOutputStream (see e.g. https://github.com/google/protobuf/issues/2365).

I've often found myself in a similar situation as myself where I just can't seem to enjoy using Make and never make it past trivial things. For that reason, I'll shamelessly plug what I think is a viable alternative: the Taskfile.

https://hackernoon.com/introducing-the-taskfile-5ddfe7ed83bd

Funny, I read that article and got excited that I finally know how to use Make. Then I looked at one of the examples he pointed to, and discovered he abandoned it in November last year :P

https://github.com/topojson/us-atlas/commit/a2a9d4279a125212...

Reply on news.ycombinator.com