Android now builds with Ninja instead of Make | Hacker News Reader

Android now builds with Ninja instead of Make

groups.google.com

98 pointsevmar44 comments

Text for those who can't/won't log in for it:

  AOSP master platform builds now build with ninja instead of GNU make. 
  The build/core makefiles and Android.mk files are read by kati 
  (https://github.com/google/kati/blob/master/README.md), which will 
  interpret them and write out a $OUT_DIR/build_$(TARGET_PRODUCT).ninja 
  file, and then the execution of the build rules will be handled by 
  ninja (https://martine.github.io/ninja/). 
  
  Building with kati and ninja provides some immediate benefits: 
  1. Faster startup time for new builds. Running kati and then ninja 
  results in builds starting about 25% faster than GNU make. 
  2. Much faster incremental builds. kati caches the mtimes of files 
  and directories it read, what environment variables it used, etc., and 
  only re-reads all the makefiles if something has changed. This 
  includes globs like $(call all-files-under). Incremental builds with 
  a warm disk cache will start within 5 seconds. 
  3. Better build command output. Ninja buffers the output of each 
  command and prints it without mixing output from multiple commands, 
  meaning error messages will be printed as a group after a build 
  failure, and not mixed with other concurrent build output. Ninja also 
  prints a configurable status line that defaults to printing the 
  percentage of build commands that have been run, the number that have 
  been run, and the number that need to run, giving a reasonable 
  approximation of the progress through the build. 
  4. Incremental builds should be more reliable. Ninja considers 
  outputs whose command line has changed to be dirty, which will catch 
  changes to global cflags, etc. and rebuild anything that has changed. 
  
  The use of ninja should be transparent, it is wrapped by the top level 
  Makefile, so all make-based commands should continue to work. 
  
  These changes are a precursor to a much larger change that will 
  replace all the Android.mk files with data files using a format that 
  provides much better error checking, even faster builds, and 100% 
  reliable incremental builds.

It didn't require sign-in to view.

Google groups, in Chrome at least, requires you to sign in if you have a google account cookie (signed in to any google account at some point), but are currently not logged in. This occurs unless you specifically run google groups in a new incognito window.

Google groups has to be one of my most hated sites. It's always so slow, so hard to read and the forced sign in thing is so annoying. I groan internally every time I get taken there.

There is some good content on there, but it seems like it's going the way of google code (which could have been github if google had played it right).

Bloggr is worse still.

At least I've never run across anything I'd want to read on Bloggr.

Thanks! This is annoying.

Not to derail further, but I just signed in, logged out, and it doesn't require me to sign in...

I appreciate the sentiment, but now I have to side-scroll to read ;-)

Maybe I'm just using an unusually large font for reading hn, but I find that <pre>/indented/code blocks on hn are best limited to around 50 character width - preferably a little less - to avoid scrollbars.

Can't please everyone ;)

if you can't get 80 columns in your viewport you need to go back to 1974 and start using 80c wide punchcards.

This may blow your mind, but I code in 40 columns. By choice. It's amazing.

only if you write in APL is this acceptable

I write in JavaScript.

Not everyone is reading on the computer all the time.

Try using phone and you will get his point.

I do use a phone. 80c wide is fine.

Two pictures: http://imgur.com/a/pu4ii

The first one has text size comfortable for reading, but it is not 80c.

The second one fits the width of the text, but is not fine by any stretch of imagination.

(The device has 5.2" display).

I get at least 109 characters in my viewport (eg: your full post above). I get about 50 characters in the <pre>-box that hn helpfully renders for code/blocks.

Colin's reply to the followup question about the last paragraph is interesting:

   On Wed, Nov 4, 2015 at 4:54 PM, Colin Cross (ccr...@android.com) wrote:
   >On Wed, Nov 4, 2015 at 4:29 PM, Dmitry Blotsky <dmitry....@gmail.com> wrote: 
   >> These changes are a precursor to a much larger change that will 
   >> replace all the Android.mk files with data files using a format that 
   >> provides much better error checking, even faster builds, and 100%  
   >> reliable incremental builds. 
   >> 
   >> Are you referring to the ninja format, or to something else? 
   >
   >Android.mk files will be replaced with Blueprints files.  For an 
   >in-progress example, see 
   >https://android.googlesource.com/platform/bionic/+/master/libc/Android.bp.

Blueprint takes build descriptions written in Go and generates ninja files from them:

https://github.com/google/blueprint/

I'm hoping this leads to increased build speeds. They really need to get rid of repo though

Wait, so they have a custom build tool that builds build scripts for their other custom build tool?

"ninja" seems to refer to this: https://martine.github.io/ninja/

Can anyone summarise the benefits of this build system vs make? According to the manual, just speed (why didn't they just work on making "make" faster?)

The Ninja manual introduction explains some of that. http://martine.github.io/ninja/manual.html

The short answer is that Ninja is conceptually similar to Make, but it dropped slow features to speed things up.

The post (quoted above in a comment) describes some of the features the Android engineers liked.

This old post from when Ninja was first released discusses "why not just improve make": http://neugierig.org/software/chromium/notes/2011/02/ninja.h...

With another four years of perspective, I think the things Ninja does could not have been added to make without forking Make (perhaps even the build file syntax), at which point there's little point in starting with the Make code. For example Ninja builds in parallel by default (like the -j flag to Make); you couldn't make that change in Make because existing users have builds that don't work in parallel.

(Disclaimer: Ninja author, no involvement in the Android work.)

You could just set MAKEFLAGS=-j<N> in the environment. I have this in my .bash_profile:

    export MAKEFLAGS=-j$((`nproc` + 1))

As evmar already mentioned, lots of projects don't support parallel make.

This paper has a lot of great information: http://aegis.sourceforge.net/auug97.pdf

Basically, the way most projects use make (recursive make) creates a whole lot of problems. This can be fixed while still using make, but if you're already revamping your entire build system, you might as well move to something that offers additional niceties like the ones mentioned in the article: better error messages, progress report, machine readable output, etc.

I like this: "Ninja buffers the output of each command and prints it without mixing output from multiple commands, meaning error messages will be printed as a group after a build failure, and not mixed with other concurrent build output."

To me, the most important feature of Ninja is that it can still track dependencies when a command produces multiple output files.

In Make, if you have a step with multiple outputs, it seems the usual thing to do is to lose track of all but one of them, and instruct the developer to run "make clean" when things stop working.

I thought Google was all about Bazel these days. Anyone know why Google is pushing two apparently competing build systems?

Only two? There are more than that. For example, Android Studio uses Gradle. Go and Dart have their own build systems as well.

It might be because Google is a big company doing many things in many languages, Bazel was only open sourced recently, and so on.

Don't forget Chromium which itself has two meta build systems (gyp and gn - similar to bazel/cmake/etc.) along with an underlying build system (ninja) which can target multiple compilers and runtimes on different platforms.

P.S. Chromium is in the middle of a transition from gyp to gn[1] which is why it has two build systems.

[1] https://chromium.googlesource.com/chromium/src/+/master/tool...

Also google doesn't give a shit about the developers that have to use the tools.

Good to hear the move away from gyp. The hand tuned build scripts (VS/nmake/make) before gyp were better.

Bazel is blaze released to the public. Android never used blaze because it had to be built by people outside Google. Different worlds.

Right, but Bazel is public, so why not use that?

because ninja launched years before blaze, and the team didn't want to switch again?

As if building android didn't need more bizarre dependencies setup.

how long until they get it to build properly i wonder? XD

sorry... just can't help but poke fun at these tools i've never needed except in the linux environment.

every time i get something from google and need to build it its a nightmare of poor build experience. good to know they are making an effort to improve things.

one click build ffs.

And how exactly do you build your projects? Msbuild? That's a 2 (4?) GB download.

I agree that the GP wasn't .. coherent. But I don't understand your comment. msbuild resides in (for example) C:\Windows\Microsoft.NET\Framework64\v4.0.30319

What would you need to download? It doesn't seem to be part of VS if that's what you were suggesting.

That version of msbuild isn't fully compatible with the one that lives in VS's folder, even when VS is installed. I've had it fail when trying to use it to compile stock projects made by Visual Studio, without anything fancy in the scripts.

re coherence there is missing context.

have you ever tried to build webrtc, for example? (its the worst case i've hit tbh, not all of google's stuff is /that/ bad)

its a nightmare. some of the dependencies are utterly horrible (systemd!!!!, old deprecated MS headers from VC6) it seems to be custom scripts called from one build system that trigger another... its below the par of what i would expect from an absolute novice.

i hear that internally its a lot easier, because there is infrastructure that downloads all these things and sorts it out for you... but from the outside world its rubbish. i'd rather have a hand crafted visual studio project, xcode project, makefile where i can do one thing and not have to understand the insides of a 'black box' library to use it for anything at all.

generally i use whatever tools come with the standard ide for the platform. so yes, msbuild, if i need to do a build from a command line for protecting my main branch or something like CI. on mac its xcodebuild, on linux i tend to use make.

i've dabbled with cmake but it doesn't solve the problems i care about the most (needing to specify the files to build by hand is a headache - but it saves me having to do it 7 or 8 times at least), for which i write my own script...

i strongly disagree with the command line tools mentality. its not the 1970s any more. this is why i say "one click" build. not "one command" build. pressing any number of keys is too much work. XD

Reply on news.ycombinator.com