Show HN: Beast – A Build System | Hacker News Reader

Show HN: Beast – A Build System

github.com

27 pointsgaurav180456 comments

Beast is a build system built for speed and power - a tool for all your build needs. GitHub: https://github.com/GauravDawra/Beast Docs: https://gauravdawra.github.io/Beast-docs/

As a project grows larger, it becomes difficult to keep track of all the build and compilation procedures that need to be followed. So what should we do??? Not to worry! Beast helps you build your projects with minimal effort and high efficiency, bringing more power to you. In addition, it is super easy to use and syntactically easy to understand, making it suitable for both: beginners and highly experienced programmers. With its new release: Nimble (v1.1.0), Beast has become much faster and stronger than before. Its build times are now overtaking or matching those of current community standards!!!

In contrast to other such build systems, Beast focusses on both: ease of usability and speed!!!

Open articleView on HN

Looks conceptually quite similar to make, or did I miss something? Did you use other build systems like e.g. cmake, meson or gn? What makes beast more usable or faster than e.g. meson?

What always amazes me: shouldn't the build system itself be as easy as possible to build (low requirements on the compiler, minimal dependencies, platform agnostic, etc.), e.g. just like "gcc -O2 buildsystem.c"? Also almost all of these systems seem to suffer from the same problems that were discussed (and solved) in the early years of software engineering, and e.g. hardly support modularization/encapsulation or static type checking. Cmake and meson are huge and complex, with a peculiar dynamic language each, not easy to install and use, and usually a factor ten bigger than what I want to build; also beast itself requires make, a recent gcc version and even flex and bison (so it doesn't e.g. run on windows, does it?).

Agreed. Essentially the build system should do whatever crackpot things it likes to bootstrap with bespoke languages and so forth, but the output should be a single somewhat auditable C file that gets fed to a compiler, not a binary with exciting dependencies on wherever it was built.

How sqlite constructs the single source/header pair looks ideal. Various efforts have bundled lua and some libraries into a single C file too.

Check in that generated C file and you have trivial 'bootstrap' for everyone else. Ideally have some CI that checks the dev files still generate that exact C file on request.

Lua is even simpler; there is an all.c which simply includes all other c files in a decent order, so you can just type "GCC all.c" resulting in a full Lua vm executable on all platforms. Other projects like e.g. Duktape offer amalgamated c files as you seem to suggest, but these are redundant.

This sounded wrong so I checked. You're correct. Current GitHub has a file onelua.c from three years ago.

Turns out I forked the project four years ago. Slightly different implementation in that I expanded the #includes to get a single file I can copy around and hacked in a few libraries (libuv, lfs and penlight iirc). Not totally pleased to discover that was so long ago.

Great to see that cc onelua.c gets the job done today.

A file called all.c is also present in Lua 5.1 from 2006 in the etc subdirectory.

Could be? https://github.com/lua/lua doesn't seem to have an /etc subdirectory and I think that's where I cloned from. Can totally believe the work had already been done by someone else and I failed to find it.

I recommend to download from the official site: https://www.lua.org/versions.html or https://www.lua.org/ftp/.

I just did an experiment with Lua 5.4.4; moved the file luac.c to another directory and then executed "gcc *.c -lm -o lua" in the src directory; compiled the vm without an issue, and "./lua" does what expected.

I'm not really sure what you are pointing at here. But if you are suggesting to have a single C file that has configurations, dependencies and commands to be run while building your project, then that is not a very good idea. Simply because.... we may not even want to use gcc to build our projects.

We may want a more suitable tool that is first of all more user-friendly (like Beast), allows you flexibility in defining custom tasks (like Beast does).

Secondly, we simply may not have gcc to compile the all.c file (as mentioned in comments below). We just want a unified tool (an CLI preferably) that does all this for us. This is what Beast does. Make and Ninja are long standing build systems that prove my point.

Plus, you get Beast binary as a standalone executable. Check it out here: https://github.com/GauravDawra/Beast/releases/tag/v1.1.0

I mean a single beast.c that contains absolutely everything required to build a static executable beast which is your build tool. Ideally in C that multiple compilers can deal with. Building that source during dev and checking it in makes bootstrap trivial for users.

The ultimate build system should be written in a language with no implementations at all, failing this it should only be buildable using itself, examples of best practice being Bazel and Gradle /s.

Well, you can expect that there is at least a compiler and runtime available compatible with the software you want to build; if your software is e.g. written in Java, then a build system also implemented in Java looks like a good fit. I usually develop in C++, so my preferred approach is to have a lean build system written in C89 which has only standard dependencies and compiles like "gcc buildsystem.c".

I don't know how you would write and deploy a build system when there is no implementation for the language it uses; this looks like an unsolvable bootstrap problem to me; you would need a precompiled build system to build the build system, and that on all platforms you want to build; and anyway I don't think that a build system language should be Turing complete (or otherwise the language should at least support modularization and static typing).

Bazel is certainly great for a lot of use cases, but it is one of those Google colossi where you have to hire a specialized team to manage and run your build. GN has a similar issue in that in practice there is an insurmountable dependency of a gazillion of Python scripts (depot tools and the like), even if GN and Ninja are written in C++ and only have standard dependencies.

Hi! You are right in saying that Beast is "conceptually" similar to Make (and Ninja). But it is much faster and much more user-friendly than Make.

Second, you are listing tools like cmake and meson. Note that these are NOT build systems but they are META BUILD SYSTEMS. Meta build systems are tools that don't really build your project themselves... instead they rely on tools like Make, Ninja and Beast to build it for you. You must have seen that Cmake emits a Makefile (or Ninja file depending upon your choice). So comparison in speed to meson will not be appropriate. (Let us just call them high level build tools)

About your third query... Well Beast is easy enough to compile. If you don't want to compile it, you can directly download the system binaries at https://github.com/GauravDawra/Beast/releases/tag/v1.1.0

I'm currently using Make to compile Beast because I originally compiled it using Make. It will soon move on to CMake. Please note that it is totally OK to use flex and bison. You don't even need them since I have already provided the cpp/h files generated by them on github!

> these are NOT build systems but they are META BUILD SYSTEMS

That's technically true, but you could just integrate the Ninja source code into such a meta build system and run it e.g. with a different command line option and then at the latest the attribute "meta" would be no longer relevant. "meta" in this context doesn't actually mean "on a meta level", but just that the tool is a transpiler from a high-level to a low-level build description, instead of a full execution engine. make (in contrast to Ninja) has features for both levels. Personally I would prefer an integrated system, so bootstrapping would not involve two (artificially) separate parts. But as I understand your message is that Beast is rather a Ninja alternative than a Cmake alternative. That's ok, although we already have Ninja and it claims to be a very fast build tool. So in the question we then just replace Cmake/Meson by Ninja.

> since I have already provided the cpp/h files generated by them on github!

Ok, I see. So it then also runs on Windows? How do you build the tool on Windows?

EDIT: actually if you're interested in GN, which has some very good concepts not seen elsewhere, I've written a tool to view and navigate GN source trees, see https://github.com/rochus-keller/GnTools. I usually use it with the Dart SDK source tree.

Hi! Even so, if you want to compare Ninja with Beast, Beast has much more user-friendly. Ninja itself claims that build.ninja files are not supposed to be written by humans (since they are not always easy to write). But Beast is super friendly here.

Moreover, the benchmarking I'm doing currently (with the Nimble release) shows that not only Beast matches Ninja's speeds, it overtakes it for different configurations and parallelism levels.

Right now it is not supported for Windows. If you have any contribution that can help in this direction they are more than welcome :)

I will surely check out your repository!

As a project grows larger, it becomes difficult to keep track of all the build and compilation procedures that need to be followed.

Anecdotally I've worked on projects that have thousands of scripts that could be built with a 5 line make file, and other projects that have a few tens of files that needed a 200 line Webpack config. I don't think build complexity is a function of project size.

Hi! The reason why you can compile such large amounts of files with only a few lines in a Makefile, is because most of these files might be carrying a pattern in their build rules (like for example most of them are to be compiled in an object file). But this is not always the case. Plus, note that I (and many others I have worked with) are of the opinion that Make may be popular but it is surely not user friendly. The syntax involved in writing a Makefile can get very messy very quickly with all the $<, $@ and what not... Beast's syntax is much more simpler and intuitive, so that everyone... (beginners or experienced) can use it

The style looks extremely similar to make, in which case it needs a good explanation of what makes it better.

It even seems to copy make's syntax, which IMO is a missed opportunity. One of the issues with Make is that the syntax is clever until you need to invoke multiple lines of shell script, like a loop. Then the tab-based cleverness becomes a pain.

Hi! Basically it is supposed to do what Make does... but in a faster and more user-friendly way. Note that Make doesn't have the best syntax. But Beast has sort of a 'python' syntax. It supports strings and integers. Variable modification assignment. It even supports scoping (within and outside beast build rules).

Plus, it is an order of a magnitude faster than Make. All in all, I don't agree with the fact that it's syntax is similar to Make. Well for one thing it has legible special variable names like 'out' and 'dep'.

So yes, while it is a low level build system, it is an improvement over Make in terms of syntax and speed.

How is it faster? And faster at what?

By 'faster' I mean better build times than Make for a project of particular size. Benchmarking is going on right now, and the difference in performance is clearly visible.

Okay, but what specifically is faster?

Make is rarely a big bottleneck, except in very pathologically written projects, and when doing incremental builds. I think the main problem with big make setups is recursive make, which prevents it from having a global view on the project, and forces the main make to execute many sub-makes recursively, all of which has overhead.

I suspect they are benchmarking recursive make vs recursive beast; the very first example in the readme suggests recursion

Good point. I have to say I'm wary of that, recursive make is not a good idea, and the problem can't be fixed by building a better make, because the issue is that it breaks up the dependency graph and creates build problems.

Eg, if you have src/lib, and src/bin, and the src/bin project depends on the src/lib library, then in self-contained, recursively invoked makefiles (or anything else), you won't get a correct behavior when a header in the library changes.

Other make replacements have solved this by making the dependency graph independent of the file structure. Hopefully beast does this too

What do you mean? make has no particular requirements regarding the file structure. You can write a single giant Makefile covering a directory tree with thousands of source files in various directories.

I meant independent of the (make-replacement)file structure. As long as the dependency graph isn't restricted to a single build file, then most of the inherent issues with recursive make are avoided.

The build process is faster. There are a number of algorithms running (on several threads) while building. That is where optimisation kicks in.

Congrats on making the release! I glanced through the docs and it seems Beast works via a graph over targets and mtimes?

I recently wrote some retrospective notes about build system design in this area that seem like they might be useful for someone in exactly your place:

https://neugierig.org/software/blog/2022/03/n2.html

Hi! Thanks a lot!! These are valuable and I am excited to take a look!! Interestingly, some of the things that you mentioned here are things that I discovered while using Make and Ninja purely through experimentation!!!

Great work!!

Increasingly, my view is that there is only room for two quite separate things:

* Task runners (rake, simple Make rules, shell scripts) etc. Basically convenience scripts for devs. * Hermetic build systems

Hermetic build systems (Bazel, Nix package manager) can be 100% confident that there aren't any secret implicit dependencies and this unlocks some really great stuff such as:

* Complete confidence in incremental builds at all time. It isn't possible for something secret to have changed * Excellent build caching, because you know all the inputs match the cache

The classic example of this is C or C++. The build step is `gcc -c main.c -o main.o`, but that build command depends on a load of header files included by main.c. Both system ones and ones local to your project. If you change a header and rerun your build rule, it only knows about main.c, knows that it hasn't changed (by hash or timestamp) and doesn't do anything. You end up having to rebuild a load of stuff just in case. Your usual system (e.g. make) has no idea about this. Various solutions exist - Beast talks about the 'get gcc to tell you about the dependencies' approach but now you've got another file to generate and manage and have in your build system, and you still can't be sure it's right.

There is a (good) argument to be made that these are problems that are only unmanageable in big software projects, and the rest of us should just take advantage of the simplicity of make (or newer better solutions like Beast). This is a similar argument to the one being made in another post about K8s and 'early optimisation'. But this is why every time someone posts a new build tool, I always hope it's going to be about 'power and reliability of Bazel' but as friendly to get going with a basic make rule.

If determined, the system headers can be elided. Bundle parts of musl with the application or work in -ffreestanding. Then it's self contained except for the compiler, which I believe some embedded systems commit to the repo along with the source.

> Hermetic build systems

Wanting to know more, I found this:

"Hermeticity: This page covers hermeticity, the benefits of using hermetic builds, and strategies for identifying non-hermetic behavior in your builds." https://docs.bazel.build/versions/main/hermeticity.html

Sounds great.

Ages ago, my teams had a policy of "one button build". Install VS C++ on a new box, open the project (from source repo), hit "Build". Tada.

We could rebuild any revision on demand. Terrific for reproducing regressions and delta-debugging.

In the Java world, with (misuse of) maven, gradle, jenkins, etc. attaining reproducible one button builds is quixotic.

For hermetic builds, everything would be digitally signed (SHA256), right? There's a spec for signing Linux kernels, which I can't quickly refind. But the idea is to apply that strategy to everything, right?

That sounds perfect.

What you basically do is take all of the inputs (source files, compiler, exactly what command you're going to run) and then produce a hash of all of that together. Then the result of that becomes the input to the next thing etc.

For caches you just say 'here's the SHA1 of the inputs and the cache server can just give you the output.

Basel does this for each build command, e.g. for each object in a big C program. Nix does it for each package.

> I always hope it's going to be about 'power and reliability of Bazel' but as friendly to get going with a basic make rule.

Have you seen https://please.build? It is very close to what I think the answer to "What if Blaze was made today?" is. It is conceptually much simpler and doesn't suffer from some of the things that need to happen in Google-scale projects/teams. For example, they haven't done the same gross thing with proto_library that google did. You only define one and it only builds the bindings that are actually needed by the targets you are building.

This looks really interesting!

Doesn't support Windows which is a bit of a shame, but maybe manageable. Will have to check it out properly at work tomorrow.

Why would I use this instead of make? I read a bit of the guide to “beast files” here and haven’t seen anything different from make other than some slight syntax differences. https://gauravdawra.github.io/Beast-docs/mainDocs/writingABe...

You should make your value proposition clear in your project README, especially if your syntax and overall model appear to be quite similar to an existing tool. Saying you focus on “ease of usability and speed” doesn’t tell me much.

I haven't used Beast, but I have used make a lot. One answer might be "just in case someone wants to put a space in a filename, or in any parent folder of where my code is".

Make does not cope with this at all. Yes I'm aware of all the workarounds and they all fall over at some point. Make works perfectly well on Windows for instance, except for the whole generation of Windows that had the users under "Documents and settings". Was almost always impossible to build code in the user home directory.

Hi! I know I haven't supported my claims in the readme. But if you see the syntax for beast build files, you will see that indeed syntax in beast is much simpler and intuitive. For example, you have strings and integers for variables. You can dereference them easily and even create new variables of any type from other variables. You can declare variables inside a build rule. All in all, the syntax is inspired from python, since everyone loves it so much. This is not all, I'm currently benchmarking Beast. The new release: Nimble is turning out to be way faster than Make and even matching/overtaking Ninja (currently one of the the fastest build systems in the community) But you are right in saying that I should include this in the readme. I will surely post this after the benchmarks

Awesome project! I'll use it for my next C++ project.

Thanks a ton :)

what languages it is for ? seems only for c/c++ ?

or it can be for any language, just put the build command line in beast file? thanks.

Hi! It is not only for C/C++, but it can be used for any project in general. This is a low level build system which runs your commands in a shell depending on the rules you define in the beast.build file. Think of it as a faster and more user-friendly replacement to Make. It's 'pythonic' syntax makes it very easy to use and it's build speeds are matching/overtaking those of community standards!

You are right in saying that since any command can be put in the beast file, it is suitable for building a project in any language.

It is a general purpose build system, with focus on speed and usability!

thanks for reply,

- examples for using it for android or iOS, or java project, and some numbers showing its better performance?

- how to use it for an android project? gradle handles it all with target(sub targets tree structure of the build tasks...), if use Beast, will still be using gradle? or the build tools gradles is calling for different things (compile, packaging,signing...)

1. Since Beast is new, there are not a lot of examples. Benchmarking is going on and it will take some time. In the mean time, you can take a look at how to write beast build files.

2. You can use it for any project you want if you know what are components of your project depend upon what other components. So as I see it, gradle is a build system for Java project in itself. Beast will not be using gradle if you are trying to compile a Java project. BUT.... you can for sure call `gradle` command from inside the beast.build file. Wherever build.gradle file is stored. So that in this case, beast provides you with an easier to use build system. Let us see an example:

In your beast file, you can have something like:

  build hello:
     ! gradle -q hello

and inside your gradle file:

  task hello {
    doLast {
      println 'tutorialspoint'
    }
  }

Hope that helps

It doesn’t make sense to have your build tool call another build tool to actually perform the build. In that case, why I would not just use Gradle directly?

True. This seems more like a general-purpose task-runner like Task [1] than a self-contained build system.

1. https://taskfile.dev/#/

Hi! Task runner would not be an appropriate term for this. Since it also defines target based on their modification times. This is what we call a general purpose build system, just like we call Make and Ninja build system.

It can do anything you want if you can provide it with proper shell commands and define targets. This can not only be used for tasks but also for physical building targets that can be built on the machine.

I believe that the project you linked here is itself also a build system and not just a task runner. But again, it uses yaml, which might not be best while defining custom things related to a build system

There can be several reasons for that. Say your project doesn't only require gradle but other tools to be run too. Or say there are some post-scripts or pre-process-scripts that need to be run. They can be done easily like:

  build pre:
     ! echo 'pre script'

  build proj:
     ! gradle hello

Plus, you might want to run some python scripts or run some other shell commands during or after you have built your gradle project.

Hi! I made a mistake in my previous reply. I forgot mentioning 'pre' as a dependency for 'proj'. I meant something like:

  build pre:
     ! echo 'pre script'

  build proj: pre
     ! gradle hello

This will ensure that 'pre' scripts are run before main 'proj' script!

Hi there, it might be a good idea to clear this up early on the github page itself. This is a good idea, but I would not read on after the first page if I don't see that it checks my boxes for what I need. And as I can see it is still not really defining it.

Thanks! Noted... I will take care of it! Thanks again

What are the advantages over cmake or meson?

Hi! I would like to point out as I have pointed out in comments above, that Beast and CMake (though might look similar) are different tools. Beast is an actual build system, that looks at your beast file and carries out the tasks you have specified. It runs at the very base level executing whatever you tell it to.

CMake and Meson on the other hand are Meta build systems. They do not run any build commands on their own. They use other build systems like Make, Ninja to carry out everything. Think of them as a fancy wrapper around build systems. Beast is here as an alternative to Make and Ninja.

Looks like a pretty neat extension of make syntax. Having better/simpler control of variables sounds great!

Thanks! That was the idea yeah!... to have more user-friendly syntax :)

Reply on news.ycombinator.com