Allowing CMake functions to return(value)
oleksandrkvl.github.io
oleksandrkvl.github.io
The CMake scripting language is just awful. It breaks with convention to do things in its own unintuitive way. (Who though the endfunction() syntax was a good idea? Or the bizarre function-call syntax?) It's error-prone, it shows cryptic error-messages, it's able to break in subtle ways, much of the documentation is for deprecated functionality, and worst of all, it lacks a consistent design pattern. One shouldn't have to think when making a Find... CMake script so that CMake can find an installed package. It should be a very boring task of adapting existing boilerplate. But it's not so. It's an agonising uphill battle with this atrocious language.
Subtle bugs like [0] wouldn't even be possible in a well-designed build-system, but CMake lets you shoot yourself in the foot as punishment for not knowing its every subtlety.
It's unfortunate that CMake is the least bad build system we have. You should probably still use CMake for multi-platform code. You shouldn't expect people to learn some obscure rival build-system just to work on your code. I'll certainly continue using CMake for what little C++ I write myself. Once you've beaten your CMake scripts into submission (generally thanks to the extensive body of knowledge on StackOverflow, in my experience), CMake works a treat on a bunch of different platforms. It's just a pity the language is so terrible.
What little C I write I compile with Makefiles, but I've seen many new C projects on Github using meson. It look better than cmake at a glance, and at this point in time isn't obscure either.
(My reaction to CMake's scripting language was unprintable).
It can compile into visual studio projects or xcode projects. That's how it achieves multiplatform support.
Basically by writing it in CMake rather then make you've automatically written your make file as well as many other platform configurations. Of course the caveat like the OP is that CMake syntax and design decisions are just plain bad.
With caveats--I've switched to Bazel. Bazel does solve the very real, very hairy problems of compiling C++ programs for different platforms, on different platforms. It is a damn beautiful thing. There are a few places where Bazel really shines. For example, generated sources, multi-language projects, and building third-party dependencies from source (without monorepos or Git submodules--you can get your third-party sources from HTTP, Git, etc).
If my project does any of these things (third-party dependencies, uses multiple languages, or generated sources), I reach for Bazel immediately.
That said--it is also impossible to give an unqualified recommendation for Bazel. The documentation and community is catching up, and a few simple things (like using pkg-config) are a bit more difficult to get working than they should.
Spaces are how Make separates items in arrays, after all. For example, take this contrived example:
foo.c\ bar.c
echo $(patsubst %.c,%.h,$@)
I went to great lengths on my recent Windows install to ensure my user profile, which has my real name in it, uses `efreak` as the name for my home folder specifically to deal with software that hates having spaces in filenames. There might well be a way to specify this in Windows, but I'm not aware of it. Instead, I had to recreate my user profile 3 times due to Microsoft syncing; first it wanted 'first last', then it wanted my Microsoft account name. I don't recall how I got it working finally; I may have renamed the account before signing into sync (it keeps my desktop environments together across several computers). Using 8.3 filenames can help too, but that requires remembering what they are.
If you've only got a single folder in the path that has a space in it, then I've had success using symlinks (or junction points on windows, since explorer recognizes those and doesn't recognize symlinks as such). Just create a folder c:\mounts or ~/symlinks to work with.
And god forbid you ever have to work with the qt build environment in Windows, extracting and deleting the source each take like 10 minutes on my ssd... I really wish it was distributed as a vhd file, or that I could convert it to one easily.
It's beautiful.
What is special about how Bazel does these? I have never had a problem doing any of this in CMake, or Meson.
In my experience the only reason to pick any over the other is because they tend to vary wildly in supporting certain compiler toolchains for non C/C++ projects.
It's not magically different than CMake and Meson, it just works in a way that I never every think "maybe make clean will fix this".
I can trust it to work regardless of the changes.
You can also, out of the box, use remote build/test execution & remote caching shared between Devs and CI.
It's also VCS aware and uses that to help prune down change sets to make builds faster.
how would that ever work if you want to build software to be packaged in a linux distro where your software has to comply with $VERSION of every library provided by the distro ?
https://groups.google.com/g/bazel-discuss/c/Ndd820uaq2U
You'd also need to make sure everything is marked as `linkstatic=false`, but I'm not sure if that would work correctly.
But, in the end of the day, you're going against the grain of bazel here. Bazel expects to statically link everything to make distribution to heterogeneous environments simpler.
More specifically, I'd like the following things happen:
1. Compile flatc with the host compiler since we have to run it later, the FlatBuffers schema compiler which compiles .fbs schemas to .cpp, just like protoc of protobuf. 2. Run the flatc on all my .fbs schemas. 3. Compile the output of 2 with the cross-compiler for the target.
With Bazel it can automatically detect the fact that it is going to run the flatc on the same box so it would be compiled with the host compiler. With CMake it seems impossible since it does not have a concept of "build host", so my current solution is to add the FlatBuffers repo via ExternalProject_Add twice, once for host configuration with a patch to change its CMake target namespace prefix to "flatbuffers_host" (because apparently this is not possible to override later) to avoid name collision, once for the target configuration, then change every reference of flatc in my project to "flatbuffers_host::flatc".
Can you elaborate? I assume those projects have to be brought into the Bazel configuration format, or does it have some support for foreign build systems?
cc_library(...)
cc_binary(...)
Some part of me still finds the idea kind of interesting. The idea of doing a Linux-from-scratch, but porting each project to Bazel in the process. It's kind of silly and would be a lot of work, but I think about being able to modify a header of any library in the system, and have that change quickly percolate up to every dependent application automatically.
The way to do it now is through bitbake, but it (rightly) is just wrappers around every other build system. The goals are a bit different than Bazel's, but it's quite slow.
But, the case where it doesn't work are way more painful than anything I've encountered from CMake + (make, ninja). Bazel is basically pathologically bad about resource use - if you run out of anything (CPU, memory, disk, inodes) you're totally hosed and probably have to reset the machine. Bazel caching is great, but boy do you pay for it in disk space - the bazel cache on my machine, for a reasonable (but by no means huge) monorepo is probably more than 100GB, and because it absolutely loves creating symlinks to everything it's very easy to run out of inodes on smaller disks even when there's plenty of space otherwise available.
Bazel basically feels like the "GC language" of build systems - it behaves like there's nearly infinite resources available and so long as that illusion can be maintained, all is well. For reference, the machines I'm using are plenty beefy (several AMD 16C/32T + 128GB ram + 1TB nvme, and several Intel 28C/56T) - it's really inexcusable that a build system makes the machine hang.
Can't you deal with this by using a mounted filesystem?
I think build systems (and packaging and installers and...) are sort of like doing your own taxes.
Things start out simple, and but at some point things become complicated and you want to just get through it and get out and forget about it until next year. (just pay someone else to do it!)
It's really too bad, I think in the software development process, there are some friction points that if they just had a little lubrication, would make everything go so much faster.
I suspect these sorts of things aren't "sexy" like machine learning, or game programming or writing a new language. But things that are important don't always correlate that way.
Seems adequate for the somewhat-de-facto standard for C++ projects!
I swear it's taken me longer to figure out how to build and include libraries with CMake than learn C. And it still makes no sense to me. Seriously, what the fuck is wrong with C/C++ package management.
I gave up on learning the build system. I recreated a working build setup in Code::Blocks UI then exported it to a makefile. Then exported that to CMake and copy pasted shit into the original build setup until it worked. Fuck it. C library management is harder than the language
I had a similar experience learning C++. Learning the C++ language isn't all that straightforward, but there are at least plenty of resources available. Build-systems felt much more like a black-box, until I'd hit them with a stick for long enough that they felt familiar.
I started out with single-file C++ programs, invoking g++ manually with the appropriate flags for include directories and linking. I think that's probably the best way to go. Work up to proper build systems a little later on.
Perhaps another good way to start off would be to just use an IDE, and ignore the build process while you're starting out.
I always feel a pang of guilt complaining about cmake, because people made it, certainly with good intentions, and it is an engineering feat... but to my mind passing over Tcl for what was created was a huge mistake. I tried to love CMake for years; turns out I’m happier writing a plain-Jane Makefile.
[0] https://vtk.org/
Scons never quite found a critical mass. I suppose it had its own quirks but I think I'd have preferred it over cmake too.
Takes 1.6s to build right now, under clang. (Amazingly, it takes 68ms with tcc, but that's a separate issue.)
If I switch to a parallel build, that time halves to only 0.8s. You can't do unity builds in parallel, because there's only one translation unit. Most people will probably get much better than 2x performance from parallel builds; I only have 4 cores.
You also can't do incremental builds in unity. If I touch just my biggest file (so only that has to be rebuilt), my compile time halves. Touch the next-biggest file, and it halves again. As a project grows, time it takes to do a full rebuild gets bigger, but incremental builds take about the same amount of time. (They may take slightly longer, as your headers grow or as the linking stage takes more time.)
You may have slightly better compile times right now, but they won't last. Incremental, parallel builds still have problems (especially when you get to the scale of linux or chromium), but but there's a reason few people use unity builds.
Unity builds also make it hard to do abstraction or information hiding well. There's a form of polymorphism in c I describe at [1] which requires separation of translation units.
1. https://comp.lang.c.narkive.com/aa5pZoND/polymorphic-oop-hac...
#define foreach(node, list, ...) for(Node* node = list->head; node; node = node->next) __VA_ARGS__
void Move(List* units)
{
foreach(node, units, {
Unit* unit = node->data;
unit->position.x += unit->speed.x;
unit->position.y += unit->speed.y;
}
}
Ah, fair enough.
(Personally, I don't generally find that the performance delta between having lto on|off is worth the compile time. Maybe for release builds. But usually performance is jut not that much of a concern; I don't bother.)
void Move(List* units)
{
foreach(node, units, {
Unit* unit = node->data;
unit->position.x += unit->speed.x;
unit->position.y += unit->speed.y;
}
}
It always amazes me how C can reach the height of most modern languages with a bit of (serious) effort.
Hell, no. If you do C or C++ (with no or a small set of dependencies) then a portable makefile is certainly easy and beautiful to write.
Autotools is the Unix-only solution to package-detection + environment-detection + build. The cross-platform solution is CMake (not counting its various obscure and immature competitors).
I do not understand the concept of "package detection".
I guess you mean about dealing with dependencies that are not copied into your code. In that case, either your dependency is correctly installed into your system (and then, by definition, it is automatically found by the compiler and the linker), or it is not. If your dependecy is not correctly installed, either the compilation will fail (which is correct behavior), or you will have to manually edit the configuration so that it can be found, by specifying its path (which is also correct behavior). Both of these things can be perfectly done with plain makefiles. No need for autotools (which is shit) nor cmake (which is shit squared).
The alternative is to have your build system to search around your disk for possible versions of the required dependency. This is a completely untoward behavior and in very bad taste.
The detection of software packages depended on by my code. Typically this means libraries, and perhaps also executables, either executables needed for the build process (such as code-generators) or executables that will be needed at runtime.
If my code uses a Boost library, the Autotools configure script (or CMake) can check that Boost is installed on the machine, check that it has an acceptable version number, and configure the build to look in the right place to find Boost's header files. If Boost is not installed, the configure script (or CMake) will fail with an error. It will do this quickly, not part-way through a build, which is poor form and can be annoying for the user.
Configure scripts can also handle optional dependencies. They can also detect platform-specific attributes and produce a header file of the values, helping the developer cope with the platform's quirks. (It might check for the availability of a compiler intrinsic, or something of that sort.)
> In that case, either your dependency is correctly installed into your system (and then, by definition, it is automatically found by the compiler and the linker), or it is not.
There is no canonical installation path for a Unix package. You're right that the linker can be expected to, well, handle the linking, but that's not the whole story.
Different distros (and different Unix-like OSs) install things (such as header files) in different places. Even within a distro, installing a package from source (rather than through the package-manager) may result in it being placed in a different directory. On Ubuntu the convention is that /usr/include holds headers installed through the package-manager, and /usr/local/include holds headers for libraries installed from source.
Of course, Windows isn't a Unix-like, and things have to be handled differently there.
The configure script should figure out that stuff automatically, the user shouldn't be given a pile of errors to manually fix. They shouldn't have to make any source-code changes just to get things to build.
Robustly portable scripts to find packages, can be surprisingly involved. [0] This is why CMake encourages us to write self-contained Find... scripts that be reused by other codebases.
> Both of these things can be perfectly done with plain makefiles
Doing so manually with raw Makefiles, isn't scalable. This is why systems like Autotools exist. The configure script handles this sort of machine-specific detail.
Large programs may have dozens of dependencies. It may be a sizeable task for the user to fix them all.
> The alternative is to have your build system to search around your disk for possible versions of the required dependency. This is a completely untoward behavior and in very bad taste.
C and C++ are not like Java. There isn't a single idealised system that you can target. You are exposed to platform-specifics, and it's on you to cope with them. This is true both at the source level (good code doesn't rely on sizeof(long) giving any particular value), and at the build level.
[0] https://github.com/Kitware/CMake/blob/master/Modules/FindIma...
> Different distros (and different Unix-like OSs) install things (such as header files) in different places.
This doesn't matter. If the distro is somewhat decent, then once a library is installed they it is found by the compiler. This is certainly true of ubuntu, slackware, voidlinux, openbsd and macos. This means that if you want to use a dependency foo.h, then you can #include <foo.h> inside your code and add -lfoo to your compilation line (inside the makefile) and everything works smoothly, regardless of where the actual headers and libraries are installed. This is true for the clang, gcc and icc compilers. If libfoo is not installed, the compilation will fail, as it should do. Everything else is unnecessary complexity. The actual directories where the files are installed are irrelevant; they are supposed to be found by the compiler (for example, through the CPATH and LIBRARY_PATH environment variables, or by the default compiler specs installed in the distro).
This is perfectly scalable. If you need twelve libraries, foo1 to foo12, then of course you need to add -lfoo{1..12} once in your makefile, typically inside your LDFLAGS variable.
Cmake is the work of the devil, it does not solve any problem and it needs to die, just as autotools.
You may wish for a situation where libraries can just be specified easily and consistently (and many languages have a package manager which does just that), but the reality for C++ is incredibly messy (because there's not one package manager but several, most lack a way to distinguish between project and system package or an easy way to add your own, and some platforms still don't have a package manager at all) and a build system which does not deal with that messy reality does not have a hope of adoption.
Cmake sucks. autotools even more so. But denying that the problem they're trying to solve exists does not help.
Regarding "a)", can you point me to a distro where after installing a library "libfoo" from the standard packaging system, the C compiler fails to find "foo.h" and the linker fails to work with "-lfoo" ? I've never found such bizarre behavior.
Regarding "b)", granted. Portability to microsoft compilers is a bitch, and may as well require cmake.
Regarding "c)", this is an impossible problem. If you install a library on ~/my/particular/path/to/libfoo, and you want to compile against this, then of course you will have to specify that particular path manually inside the compilation system. This is possible with cmake, and there's even a beautiful user interface for that. It is just as easy with plain makefiles: you just add the particular path to the C_INCLUDE_PATH and LIBRARY_PATH variables (they work exactly as PATH). No need to mess with -I nor -L. Notice that here you are not using any of the cmake "detection" capabilities, you are just specifying the path manually. This is correct behavior, of course you would not like cmake to spy on its own inside your home dir to see if it can find a certain dependency.
> You may wish for a situation where libraries can just be specified easily and consistently
Yes I do. And I'm very happy that this is the case.
> but the reality for C++ is incredibly messy
thanks to autotools, cmake, and other tools written by people who do not understand the concept of PATH. The reality is easy, not messy, it's just people who make it messy intentionally; more out of ignorance, not of malice.
Look for example at the imagemagick "detection" rules shown by the OP as a pinnacle of useful and complicated thing that cmake does. It is completely unnecessary bullshit! If imagemagick is installed, then these executables are already on your PATH, and they are found automatically by the shell (that's exactly its job!). If imagemagick is not installed, or if it is installed on a secret directory, then there's no reasonable hope for cmake to "detect" it (other than scanning the whole disk, which I hope it does not try to do...); and if you want to use this secret installation of imagemagick you will have to specify its path manually somewhere in the compilation system. For example, by setting a PATH variable inside your code.
Everything that you say is right, and it reinforces my conviction that "detecting" installed libraries never needs to be done. Yet you have hordes of idiots writing detection rules for cmake. What a time to be alive.
To return to a point I made earlier, part of the value of having a configure step before the build step, is to handle all the platform-specific stuff early, failing early and with good error-messages if something is missing. I've seen builds fail several minutes in, it's very annoying, and the configure step should make all the checks necessary to prevent this happening. If the configure step succeeds, I should be able to rely on make succeeding.
You make a good case that this step should really be very simple, and I agree, especially considering pkg-config exists, [1] but here we are. (True to form, CMake is able to leverage pkg-config, but makes it painful and confusing. [2] You have to be a master of CMake to figure it out, and the correct answer depends on the CMake version, despite that both pkg-config and CMake were first released in the year 2000.)
Also, optional dependencies can be handled properly. I don't see how you could easily handle this with pure Make.
[0] https://github.com/Kitware/CMake/blob/master/Modules/FindBLA...
I wonder why CMake cannot embed an alternative language? You could have CMakeLists.lua files modify the same targets and properties using Lua.
CMake scripting is bad. Fragmentation of build-systems is probably even worse. Like I said, it still makes sense to use CMake for multi-platform code.
If the CMake folks made more of an effort with documentation, many of my complaints would go away. It's not necessary to burn down CMake and start afresh. The awful language wouldn't matter so much if there was clear and thorough documentation with plenty of complete examples, and in particular, definitive guidance and reusable boilerplate for the most common jobs (e.g. package discovery).
> I wonder why CMake cannot embed an alternative language?
The last things CMake needs are even more complexity and even less consistency.
I honesty have no idea how some people come up with these absurd comparisons. I mean, who in their right mind feel is appropriate to compare a Makefile with a build system? And how clueless about things are "in the Linux world" do you need to be to claim that Makefiles are how people build software in Linux?
I mean, I've been using Linux for a couple of decades now and I can't remember the last time I saw a project that relied on Makefiles to build. At best projects use some kind of Makefile generator. The ancient ones use autotools and even those are going the way of the dodo.
> CMake allows us to handle almost all those things right except one - it doesn’t have return values
This is a red flag. If you're hoping to write functions with cmake... You're doing it wrong. Very wrong. You add targets, and you add rules to the target. If you find yourself wanting to do anything other than this then you should take a step back and read a modern cmake tutorial again, because odds are you're doing it wrong.
Makefiles, at least in my corner of the industry, are exceedingly common for projects with a build process of medium-to-low complexity.
Ideally the official Find... packages should not make use of deprecated CMake functionality, they should be actively maintained to follow the current best-practices, so that they can serve as examples of the right way to do it.
As I rambled about in my longer comment yesterday, this kind of work shouldn't require you to think, it should just be an exercise in applying boring, samey, dependable boilerplate.
^ Is that a good one?
Then again, if your build should output a .nupkg, the whole infrastructure seems to be lacking. You are left alone doing things that seem like they should be part of a package manager. e.g. handling of the version number. The whole build system knows the build output, but you have to specify it by hand again, in a different file.
I don't know what GNU/Linux you're using, because in my experience, most packages going into a distro use make.
Oh, and then there is that whole denominator in the GNU/Linux fraction that's built with GNU Make.
Maybe you don't see Makefiles because you're relying on binary packages?
no, they use autotools or cmake, which happens to generate makefiles. the package authors themselves definitely don't write any makefiles by hand except maybe for trivial packages (and frankly I'm not sure)
Also note that not all Autotools-based projects use Automake. Using only Autoconf, with a hand-maintained Makefile, is a thing.
The goal of Autotools is to produce a software distribution that doesn't require the end user to have Autotools. The end user has a shell to run the configure script, and that's it. If that works, they run "make".
About the most sophisticated generation the user will see is that some Makefile.in template is passed through sed to substitute @@XYZ@@ variables.
And coming back to Autotools, some of the largest Autotools projects (see the GNOME ecosystem) has been moving/moved to Meson, which can also be used to generate Ninja rules instead of Makefiles.
CMake Generators: https://cmake.org/cmake/help/latest/manual/cmake-generators.... Meson: https://mesonbuild.com/ Bazel: https://bazel.build/
GNOME and Meson: https://wiki.gnome.org/Initiatives/GnomeGoals/MesonPorting
That's what all build systems do. Hell, cmake's description is "cross-platform Makefile generator".
The rationale is quite obvious: the DAG part is solved problem. The hard part is extracting platform-dependent info, perform validations, and use it to generate the DAG.
And a Makefile is a DAG.
I've gotten rid of several of our pre modern cmake functions, but more remain.
so how do you propose to do code generation, parsing stuff like /etc/lsb_release to work around a bug for people building under ubuntu 16.04, or that kind of thing ?
In my last 3 professional C++ projects I’ve actually used Rust’s cargo tool as my build tool for C++ code. Infinitely better than CMake, plus you use Rust as your scripting language for your build system, which has great libraries for it like the cc crate, and pretty much a crate for whatever it is you want to build (eg ispc? no problem).
CMake has a terrible scripting language, but its backends are very good. Being able to generate Unix makefiles when working from GNU/Linux, and then generate Visual Studio solution files when working on Windows, is pretty great.
One of my former bosses once quipped "CMake is confusing until you realize it's just a shitty version of BASIC that only has global variables"
go build .