Rewriting Libimagequant in Rust for Portability | Hacker News Reader

Rewriting Libimagequant in Rust for Portability

pngquant.org

196 pointstodsacerdoti76 comments

Kornel is also behind the DSSIM library (in Rust), which I've found useful for comparing images before and after compression.

https://github.com/kornelski/dssim

Fun fact: I wrote DSSIM as a development tool specifically for pngquant, because I needed a way to measure improvements in dithering. Classic per-pixel algorithms only saw it as added noise.

And the excellent Mac OS image compression utility ImageOptim: https://imageoptim.com/mac

> Trying to make the library more object-oriented exposed a "drum-stick" design issue: does play(drum, stick) translate to drum.play(stick) or stick.hit(drum)?

In cases where OO doesn't make sense, the solution is to simply provide a functional API instead. Rust stdlib has plenty of such APIs, it's fine to have free-standing functions!

Agree, there's no rule that everything has to be a class in C++.

I suppose drum.play() could default to using a stick.

I think the GP's idea is to just do play(drum, stick) as its own function and not tie it to either class or have it be a passthru into internal methods for either.

You can even make everything a class, and still provide a functional-ish interface, by using the class as a namespace with a common immutable context.

But...that was exactly the question the article had. Should drum be the enclosing class, or should stick be? What he cares about is the verb play, but he has to encapsulate it in a noun, and none of the ones actually involved dominate so clearly as to moot having to make a choice.

  musicPart.play(drum, sticks);

Basicslly you can replace the module namespace with a class namespace if desired. What module could that be?

Right. But that's just it. "musicPart" is an object. A noun. What the hell is a "Music Part"? It's a completely synthetic object you've had to create solely to namespace functions. It's like how so many large Java projects have static "Utils" classes.

What you've done is not OO; in fact, it's an anti-pattern in OO. You've created, as you correctly note, a namespace or module around static functions; there is no meaningful encapsulation of data.

My point isn't that it can't be done, it's that trying to follow best OO practices, you run into this sort of problem (which has had a lot of digital ink spilled on it). The fact there is a pragmatic solution isn't in any doubt; it's that such a solution intentionally eschews the data encapsulation that OO demands (but not imperative or functional approaches, which instead say it's the right approach to do that).

If you ever played music, you know it consists of parts. Say, a percussion part, a bass part, a rhythm guitar part, a lead guitar part, a vocal part, etc.

Why would 'play' hang off of the (presumably static) "MusicPart" then, rather than, "Player", "Musician", "Composition", etc?

Also, you realize that this whole discussion is proving the point, right?

But now the traits aren't connected with drums (or sticks) but MusicParts. Or you end up having some `drum.foo` too.

> using the class as a namespace with a common immutable context

This is why we can't have nice things.

Pray elaborate.

I guess he is talking about traits. How do you 'functionize' them?

I wasn't thinking about traits when I wrote that, but you make a great point — that's a whole another way to design things. Maybe it could have been something fancy like:

    impl<T> Sound for T where T: Percussion<Accessory=Stick>

There's also the pattern used in from+into where one implementation is wrapped to provide its counterpart on the other type. Nice to read but feels a little extra magical for someone without a lot of Rust background.

Blanket implementation is a very common idea in Rust. The standard library "owns" its own Traits and so it gets to provide blanket implementations for those, but you can do the same for your own.

Here's (really, this is literally the code) how std provides a blanket implementation of Into given From:

  impl<T, U> Into<U> for T
  where
      U: From<T>,
  {
      fn into(self) -> U {
          U::from(self)
      }
  }

There's actually a chain, From implies Into, Into implies TryFrom, then TryFrom implies TryInto.

I think this is an excellent example of why people get so excited about Rust. It can't do anything you can't do in C or C++, but it makes a whole bunch of things significantly easier and less stressful. Which makes maintenance easier, and in practice often leads to performance improvements because more ambitious designs can be attempted.

> I think this is an excellent example of why people get so excited about Rust. It can't do anything you can't do in C or C++, but it makes a whole bunch of things significantly easier and less stressful. Which makes maintenance easier, and in practice often leads to performance improvements because more ambitious designs can be attempted.

For most people Rust is just overkill, what I really want is secure subset of c++, enforced by compiler with hardened standard library(that is sanctioned by standard body, might be not 100% compatible with generic one, because it contains only a subset of functionality and some functions might work differently).

To me "C/C++, but secure" is equivalent to dreaming about sex on a beach. Sounds lovely as a concept, because you imagine the positives, not where the sand will get in.

It would be nice to keep the same language, same libraries, and codebases, and just somehow make them safe. But these were written for C++, not for a restrictive subset. You'll end up removing features that dependencies use (e.g. can you even ban non-smart pointers?). You'll add runtime checks to constructs that existing code assumed were free, and never needed to avoid (zero cost is a reason to use C++, otherwise a GC+JIT language can have runtime checks, but better). You will have to rewrite your code to clarify lifetimes, ownership, and remove constructs compiler can't analyze for safety.

You get many of the same pains that switching to Rust has, but none of Rust's features, and not even a fully-functional C++.

A subset of C++ cannot simultaneously be secure and useful, because to be secure a subset would need to ban pointers, references, iterators, moves and the ability to include code that is not part of this subset (for the last point modules might help, but they aren't here and they introduce their own ODR violations cases, so "safe" is a stretch even for them).

To retain usefulness, we need a superset of C++ that would include either a GC (at which point, why use C++), or lifetime-like annotations, and the need to actively migrate the ecosystem to it. At which point what you have is a rust-shaped language with an outdated compilation model, tons of pitfalls, an inexpressive type system, and a slightly better interop with C++.

Considering that such a language doesn't exist, that which features to remove and which to add will be source for endless discussions, I'd say that the move that isn't overkill is to use the language that actually exists today and delivers on these promises. Rust isn't overkill, despite 10-years of C++ experience I'm way more productive in Rust, be it while writing code, fetching dependencies, reviewing code, and of course all of this compounds during maintenance. You do need to invest 3 months to learn the language, and some more to write the occasional bindings, but they will be very quickly recovered by avoiding writing CMake (not even counting asan/valgrind debugging sessions).

> what I really want is secure subset of c++, enforced by compiler with hardened standard library(that is sanctioned by standard body, might be not 100% compatible with generic one, because it contains only a subset of functionality and some functions might work differently).

Problem with this is, that your code would use such subset, but what about libraries you use ? Then you would have confusion as to which libraries are safe to use in such new C++, would they work with the standard C++ ?

You would effectively have 2 languages inside of one. Remember all the C/C++ confusion and how compatibility to C has kept it back? That would lead to similar problems.

I'm not sure about this. Remember that most people (even most software developers) don't know C++ (or even C). And IMO (as someone who has tried to learn both) Rust is a lot easier to learn than C++

Careful now, you'll get accused of being in the Rust Hype Squad.

Snark aside, this is what Rust proponents like myself have been trying to say. Rust isn't some magical panacea to all of the problems we face in computing, but it does solve some frustrating and endemic problems outright while making others significantly more tractable. And that it manages to do so without sacrificing execution performance (though at a generally reasonable cost in compilation times) and keeping code complexity more manageable than equivalent C/C++ code is just icing on the cake.

What the language adds in complexity—in my experience at least—reduces the complexity of my own programs and makes it much simpler to write them and reason about them. There's definitely a learning curve as the language "wants" you to structure programs a certain way while it takes time to build up the intuition for doing so. But having done that, I've been able to apply those lessons to my projects in other languages to great success.

It's funny because I've been using pngquant for optimizing assets we display in our Rust video game oxide extensions(by facepunch).

We get closer and closer to getting Rust re-written in Rust every day... hahaha

This is one of those rare situations when monoculture is superior. Integrating multiple ad-hoc build systems is a nightmare, especially when cross compiling. Using cargo and rustc is a dream by comparison. It would benefit the C/C++ community to rally around a single build/compilation tool chain. Excellent write up!

That's the kind of analysis I want to see. Really good write-up.

I am excited about rust supporting more platforms so that this kind of thing can be done more widely without sacrificing another kind of portability.

Lots of extreme generalisations and exaggerations of the situation with C. It's great that you like rust and want to write your code in it, I don't really care at the end of the day, but vague statements such are not very useful or productive.

>There's no standard for building C programs.

Is there a written standard for compiling rust programs? I don't think so. Rust doesn't do standards after all.

Jokes aside, this is basically irrelevant. Why does there need to be a standard for building C programs? Making something build across platforms (including cross compilation to other platforms) has been solved by everyone and their grandmother at this point. This means that aside from taking the route of re-implementing it yourself (not particularly difficult) you have literally dozens of perfectly good options to chose from.

In fact, most of this rant sounds like it stems from your dislike of OpenMP not C.

>I could have kept the library in C and replaced OpenMP with some other solution instead, but I wasn't keen on reinventing this wheel. Even basic things like spawning a thread run into C's portability woes.

Thankfully, you don't need to reinvent the wheel, the problem of cross platform multiprocessing in C has been solved at least a dozen times by now. It should be trivial to use one of these many options (or even write your own, even if you don't like the idea of "reinventing the wheel" it's certainly not particularly difficult either and can likely be lighter than any other more general option).

>The platonic ideal portable C exists only as a hypothetical construct in the C standard. The C that exists in the real world is whatever Microsoft, Apple, and others have shipped. That C is a mess of vendor-specific toolchains, each with its own way of doing things, missing features, broken headers, and leaky abstractions. Shouting "it's not C's fault, screw <insert vendor name>!" doesn't solve the problem, but switching to Rust does.

I mean, rust is hardly helping you here. There's one implementation of rust doing their own thing so far, nothing stops anyone from making this situation as "bad" as in C. You may as well just say you only support clang or only support GCC and base your project around that assumption. In fact, if you're making the mistake of trying to support a microsoft compiler then I would argue that you've made your life difficult for no reason and blamed it on C.

>dreadful state of C dependency management did.

Hardly any more dreadful than rust's solution to the problem.

Let's say that, as you recommend, I only support one compiler and one build system in my C project. What is the "easy" solution to the problem stated by the author ?

How do I add multi-threading to my project in a way that allows me to run a function on all available cores in two lines of code, while making the project cross-compile to every major OS and architecture from an M1 mac ?

In rust the solution is just to edit a file named "Cargo.toml" and add one line in it: rayon = "*".

>rayon = "*".

Doesn't this either effectively force you to pin versions to ensure your code doesn't break or require you to constantly re-test your code to ensure there wasn't a breaking change? I don't see how that's a desirable thing.

>How do I add multi-threading to my project in a way that allows me to run a function on all available cores in two lines of code

The author had to rewrite the project in a different language to add this cross platform multi-threading support. I'm not saying you can do this in two lines of code, the code likely needs some modifications and some considerations, but it's definitely easier to just use one of the multiple cross-platform threading libraries available (or even to write your own) and adapt the code to use it than it is to rewrite a whole project in rust.

It's a little bit disingenuous to suggest that in rust any parallel processing task in any project out there can be achieved with 2 lines of code, it's like claiming that OMP can be effectively used by any C codebase with just a 1 line #pragma. If you write your code with the intention of using rayon for your multi-threading tasks then obviously it will not require any modification to make it work with rayon because presumably you had already written your code in a way that makes it usable with rayon. If you write your C code with the intention of using it with your multi-platform threading library of choice then it won't require changes to make it use that library.

> Doesn't this either

Cargo pins versions via Cargo.lock for you automatically. It's still not a great idea, (because when you update dependencies it will go to whatever version is the latest, which may not be compatible) and crates.io disallows packages to be uploaded that use this, but it is the easiest built-in way. If you install cargo-edit, you could "cargo add rayon" and it will add it with the latest version specifier instead, which is slightly easier and better but isn't built-in (yet, someday...)

> The author had to rewrite the project in a different language to add this cross platform multi-threading support.

They're not saying that, they're saying "how do I do what Rayon does for Rust in C?" OpenMP is sorta kinda an answer there, but the post is claiming it's not as good of a solution. "Rewrite it in Rust is easier than using OpenMP" is not what's being said, and you're certainly right that porting to another language entirely is not simple.

> If you write your code with the intention of using rayon for your multi-threading tasks then obviously it will not require any modification to make it work with rayon because presumably you had already written your code in a way that makes it usable with rayon.

The claim being made is that due to things like the borrow checker, this is often true for Rust code generally, moreso than it is with C and OpenMP. And, in Rust, if it doesn't Just Work, you'll get an error message pointing out exactly why not, whereas (in my understanding, I haven't used OpenMP) that is not going to happen in C.

> The claim being made is that due to things like the borrow checker, this is often true for Rust code generally, moreso than it is with C and OpenMP. And, in Rust, if it doesn't Just Work, you'll get an error message pointing out exactly why not, whereas (in my understanding, I haven't used OpenMP) that is not going to happen in C.

Firstly, nobody knows what claim is being made here other than the author of the original question. Maybe let's let lovasoa reply.

Secondly, no, unless rust is the most rigid language ever invented (which as far as I am aware it's not) there most certainly IS a way to write code in a way where it may need to be modified to work with another library. I highly doubt that literally any random code unit in a random codebase could be made instantly parallel with no code changes other than calling it using rayon, this is just wishful thinking. Even not considering fundamental problems like figuring out how to chunk a workload appropriately for parallelization. The point I was making is that for any given codebase, it's possible to write code in such a way that it can be used with some tool easily such that it only takes two lines to make use of that tool, just like it's possible to not do this and require rewriting things.

You can't make judgements on how easy to use a library is on the basis of the fact that someone who rewrote their code specifically to use that library was able to use it by changing two lines of code. With OpenMP it certainly can sometimes be as easy as adding a #pragma omp parallel for somewhere, or it can be much more difficult than that. Nothing stops you from engineering a project to specifically be tailored towards working well with a particular library.

For me it was easier, or at least more pleasant, to rewrite the project in Rust than to deal with C build issues for various platforms.

It's not only that one thing I've got stuck with. I've been maintaining this project for years, and there have been endless issues with the build: GNU vs BSD make incompatibilities. grep doesn't support the same flags on all systems. -o /dev/null does not work in msys. Flags for MSVC, GCC, and ICC flags, including odd ones like aligning the stack so it doesn't crash on SSE loads. The arcana of static linking with compiler-private runtime libraries, for multiple compilers. __declspec. Windows charset woes. MSVC<>MinGW incompatibilities. All the combinations of uname -m. All the missing headers. pkg-config sysroots. libpng.pc is sometimes sabotaged to force use of libpng-config. Moving -lm around. Apple clang has SDK includes separate from include paths. macOS system frameworks pulling in duplicate incompatible versions of libraries I'm trying to use. .h and .so ABI mismatches.

Why are these my problems? I just want to process some pixels.

And regarding Rust parallel code working out of the box, it really does happen. Rust's single-threaded iterators already require a high degree of isolation. The borrow checker already has shared XOR mutable rule for all code everywhere. Mutable globals are already banned. So yeah, it often is as easy as changing iter() to par_iter(). Best of all, when that is not safe, the compiler will show you exactly where and why, even if it's due to dependency of a dependency in some deeply nested function call.

>GNU vs BSD make incompatibilities

... just specify which one you intend people to use? both are about equally as portable

>-o /dev/null does not work in msys

weird, sounds like an msys bug

>Flags for MSVC, GCC, and ICC flags

If you're going to switch to rust to avoid having to deal with MSVC and ICC, why not just take a shortcut and simply switch to only supporting gcc and clang?

>The arcana of static linking with compiler-private runtime libraries

I'm not sure what you mean here.

>__declspec

Just don't support MSVC.

>Windows charset woes

Can you elaborate?

>MSVC<>MinGW incompatibilities.

I don't need to repeat myself.

>All the combinations of uname -m.

Can you elaborate on why you would use uname -m instead of instructing the person building the software to configure the build correctly for their system? (it's not hard in my experience)

>All the missing headers. pkg-config sysroots. libpng.pc is sometimes sabotaged to force use of libpng-config.

Can you elaborate?

>Moving -lm around.

See note above relating to uname -m.

>Apple clang has SDK includes separate from include paths. macOS system frameworks pulling in duplicate incompatible versions of libraries I'm trying to use.

Can you elaborate?

>.h and .so ABI mismatches.

That sounds like a problem completely unrelated to you.

>Why are these my problems? I just want to process some pixels.

Genuinely I don't know. I've never heard of half of these problems and I've dealt with code which has had to build for a lot of systems. I target my own projects for windows, linux (many various distros including niche things like nixos and alpine), OpenBSD, FreeBSD, NetBSD, and Mac and have never had these issues. I am not necessarily saying there aren't problems, but a lot of it sounds like problems which could be avoided without switching to a different programming language.

>And regarding Rust parallel code working out of the box, it really does happen. Rust's single-threaded iterators already require a high degree of isolation. The borrow checker already has shared XOR mutable rule for all code everywhere. Mutable globals are already banned. So yeah, it often is as easy as changing iter() to par_iter(). Best of all, when that is not safe, the compiler will show you exactly where and why, even if it's due to dependency of a dependency in some deeply nested function call.

Do rust programmers never write for loops? How does this work for situations where you're performing an operation until a condition is met? What about iterators which return more elements than you have ram? How does the work get split when the work elements are non-uniformly difficult? There's so many questions here... I still don't see how it's possible to have a parallelization library magically able to parallelize everything with only one tiny change.

> Can you elaborate on why you would use uname -m instead of instructing the person building the software to configure the build correctly for their system? (it's not hard in my experience)

It's also not hard in my experience, but mysteriously, it seems to be hard for my users. This creates endless support load for you. Yes, "those users should not use my software" is a valid choice, but if you want to support those users, automatic build setup is pretty much necessary.

I too wish my users would just follow the instruction which is easy. Unfortunately, wishing does not solve the problem.

> How does the work get split when the work elements are non-uniformly difficult?

This is handled automatically by rayon using work-stealing. Unbelievable, I know. Intel TBB does the same for C ecosystem by the way.

> Can you elaborate on why you would use uname -m instead of instructing the person building the software to configure the build correctly for their system?

I think this is indicative of why we disagree about these things being a problem. I would like "make" (or whatever single command) to just work on every platform out of the box, without users having to manually follow a set of instructions to avoid builds breaking. It's not pure idealism — when builds don't work, I get unsatisfied users who ask for support or give up on my library. Having builds work painlessly solves both of these problems.

"cargo build" is the closest I can get to this ideal, with the only manual step being to install Rust. Rust does smooth out more than requiring even a single C compiler implementation, because it also handles dependencies and abstracts more OS APIs (POSIX not required).

I find it baffling that in C it's accepted as normal that builds can break on trivialities like "can't find foo.h", "__turblb7 is undefined", "unknown flag -florbz", that users need to perform manual steps to fix these problems for themselves, and that for non-Linux these steps are often just "lol, good luck with that". It's not about being hard per se, it's about wasting time on fixing brittle capricious software. Rust/Cargo shows it can be easy and reliable, and all of those "set up your system correctly" steps did not need to exist.

----

> Do rust programmers never write for loops?

Basically yes. There's no `for(;;)` loop in Rust. Rust's `for` consumes an Iterator object, and people prefer iterators because they avoid bounds checks compared to code using [] indexing.

> performing an operation until a condition is met?

There's take_while(), all(), any(), and similar iterator methods.

> more elements than you have ram?

Iterators are streaming elements one by one. Memory usage suddenly multiplied by the number of cores is a legit problem, but it's universal, not specific to Rust.

To be clear, there are problems that don't parallelize at all. There are loops that need a few more changes besides adding "par_". But the point is that much more often than in other languages the code is already very parallel-friendly.

That's because the borrow checker strongly discourages shared mutable state. There are no concessions for single-threaded programs (e.g. all globals must always be mutexes or atomics, even when the program does not use threads). Standard idioms are designed to work in parallel contexts (Send/Sync compatibility is a property of every Rust type). So even when you write "single-threaded" Rust, you already do most of the work required to have it parallel, so then the step to running it in parallel is relatively small.

I get the incredulity, but it is not wishful thinking. It really does work that way. Given a random Rust codebase, it is more likely than not it would parallelize with rayon, cleanly and trivially.

Of course it is possible to write Rust in a way it wouldn't parallelize with rayon. But this is an empirical statement: a random Rust codebase isn't likely to be written that way.

I disagree about dependency management. Let's say, to avoid reinventing the wheel, I decided to use GLib for cross-platform threading. How do I add GLib as a dependency to my C project? Now compare it to adding Tokio as a dependency to my Rust project, which is entirely trivial.

> How do I add GLib as a dependency to my C project?

You document in your documentation (or some metadata file format of your particular liking) the fact that your project requires GLib and which versions of GLib it supports (hopefully GLib supports semver). Then you take that version, feed it to pkg-config with the right version information and feed that into your build process. Depending on your build setup it may be as simple as putting it in your cargo.toml. Unlike with cargo, you also get to chose as a person compiling the project whether you want the code to statically compile a version of your choosing or just link against whatever your distribution provides.

Your first two sentences:

> You document in your documentation (or some metadata file format of your particular liking) the fact that your project requires GLib and which versions of GLib it supports (hopefully GLib supports semver). Then you take that version, feed it to pkg-config with the right version information and feed that into your build process.

refute the subsequent claim:

> Depending on your build setup it may be as simple as putting it in your cargo.toml.

It isn't as simple, cause you have to do more.

This glosses over support for operating systems that lack pkg-config, so in practice it is more complicated than that.

>It isn't as simple, cause you have to do more.

No? "or some metadata file format of your particular liking" is the sentence which includes "depending on your build setup it may be as simple as putting it in your cargo.toml".

Unless you disagree with the whole concept.

More importantly, adding dependencies doesn't need to be that easy given that you should focus on not having very many of them in the first place. This doesn't change the fact that it CAN be that easy.

>This glosses over support for operating systems that lack pkg-config, so in practice it is more complicated than that.

There's one system which doesn't really natively integrate well with pkg-config, that's windows, for windows the build instructions can simply be: cross compile from another operating system or build using a unix envirionment for wibdows. It's not that hard and for the average joe not willing to do this it's absolutely trivial to provide precompiled binaries.

> How do I add GLib as a dependency to my C project?

Use symbol versioning.

Anyway, I feel the author is a bit unfair. He traded supporting multiple compilers for Rust's only and it is not really surprising that he felt that the maintainer burden decreased.

As soon as Rust gets broad adaption I guess there will be subtly incompatible compiler vendors in Rust too.

> Use symbol versioning.

That doesn't mean anything.

> Anyway, I feel the author is a bit unfair. He traded supporting multiple compilers

They wanted to support multiple platforms, that multiple platforms implied multiple compilers was part of the portability issues the original goal led to.

> They wanted to support multiple platforms, that multiple platforms implied multiple compilers was part of the portability issues the original goal led to.

No, people are just inflicting that upon themselves for no good reason. I ship software built with the same clang / libc++ toolchain targetting Mac / Windows / Linux and things works absolutely fine.

> I ship software

Is it an open-source library?

> That doesn't mean anything.

I meant that you can specify symbol versions to link to glibc specifically.

https://sourceware.org/binutils/docs/ld/VERSION.html

> They wanted to support multiple platforms, that multiple platforms implied multiple compilers was part of the portability issues the original goal led to.

He could have used gcc on different platforms too. He writes:

"the platonic ideal portable C exists only as a hypothetical construct in the C standard. The C that exists in the real world is whatever Microsoft, Apple, and others have shipped. That C is a mess"

I.e. his problem could have been solved by sticking to one compiler, be it Rust or Clang ...

> I meant that you can specify symbol versions to link to glibc specifically.

You're confusing glibc and GLib...

> He could have used gcc on different platforms too.

That certainly would have solved the multiple platform issue, I'm sure gcc has great and easy support for producing VS-compatible dlls, and that there's nothing an ios dev likes to hear more than "you can use GCC to compile libimagequant".

> I meant that you can specify symbol versions to link to glibc specifically.

sanxiyn was talking about GLib, not glibc, and these are two very different projects...

I don't think that fragmentation is a given.

Other implementations may never take off. Python, Node.js, Java, PHP, C#, Golang, Ruby are all more popular and older than Rust, and did not develop fragmentation like C. They have one dominant target, and other implementations have to follow it, or die. Rust's situation is closer to these languages than to C.

But even if there were serious contenders, Rust has already established a strong baseline: there are over 70,000[1] packages that require Rust+Cargo in their current form. A viable alternative has to support them, or it will be niche like mrustc is.

[1]: https://lib.rs/stats

Ye that is true (isn't Java a bad example though with Google-Java, OpenJdk vs BigEnterpriseJdk etc?).

Java could have been a good example, but Sun had a rather strict validation process for calling something Java.

Furthermore, there are big difference in philosophy with C:

1. IB and UB are not considered normal parts of specifications, meaning there's way less opportunity for originality in the interpretation of the specifications

2. there tends to be an ur-implementation, and notable divergences from that tends to be interpreted as either a bug in the other implementation(s) or a lack of specification to be resolved between all implementations

Rust only has UB in unsafe (AFAIK), which greatly limits implementation flexibility in terms of observable behaviour; and the reference implementation would very much be considered the reference implementation, so I expect e.g. rust-gcc will be sticking close to the reference implementation and behavioural divergence will either be fixed to match, or will lead to more precise specification and both implementations converging.

Probably eventually with, if not a Sun-style validation suite, a Ruby-style Spec Suite (https://github.com/ruby/spec).

You use cmake,vcpkg or conan, or better yet C11.

Setting up CMake to support dozen hardware architectures and operating systems (and cross compiling), have targets for basic stuff like code formating, linters, running tests with/without valgrind, sanitizers, etc. generating code coverage instrumentation, etc. used to be a huge time sync.

For projects using CMake, I'd argue that one of the main features of Rust is that it enables to spend all that time into adding actual functionality to your programs.

I doubt that cargo covers all those scenarios as well, given how Google and others have gone implementing their own replacements to cover exactly such kind of workflows.

> I doubt that cargo covers all those scenarios as well

* cross compilation: cargo --target=... , you can also cross compile and then run tests for a different target easily (by running them with qemu or in a VM, or via USB in an actual phone or embedded device, etc.)

* formating: cargo fmt

* linters: cargo clippy

* valgrind: cargo valgrind

* sanitizers: RUSTFLAGS="-Zsanitizer=..." cargo test builds and runs tests with sanitizers

* code coverage: cargo tarpaulin

and that's just for the basics.

I've seen teams spending weeks on CMake and only being able to set a subset of these up.

There are many others, like dumping the inline assembly of a function in a big project (cargo asm function_name), dependency management, etc.

As an ex CMake developer and CMake user for >10 years, I'm happy to work on Rust full time just for the benefit of not having to use CMake ever again.

>I've seen teams spending weeks on CMake and only being able to set a subset of these up.

Aside from the fact that formatting has nothing to do with building (or development for that matter) that sounds like either a CMake issue or an issue I described in another comment where a problem is identified, a solution is hypothesized, and an attempt is made to implement the solution, all without ever considering if either a: the problem identified is a problem or the right problem, b: the solution hypothesized is the only or best solution and c: if the implementation is implementing the solution or not.

>As an ex CMake developer and CMake user for >10 years, I'm happy to work on Rust full time just for the benefit of not having to use CMake ever again.

There are other options than CMake.

I've shipped large C projects to dozens of platforms and have encountered all the issues the author mentions and many more.

You claim that these issues aren't "real".

Even if you are only shipping a project nobody uses to a handful of platforms, you'd have encountered many of them.

So IMO the only conclusion I can draw for your comment is that you haven't even done that, and therefore you have no idea what you are talking about.

Which is why all your comments in this thread are downvoted.

Every single one of them makes your ignorance abundantly clear to everyone in the community.

The only conclusion I can draw from what you just wrote is that you've encountered easily fixable issues and rather than try to solve them in a sensible manner you've backed yourself into a corner of solving them in a complicated and difficult way.

In my experience in the software industry it's not uncommon for teams of people to persuade themselves that there's no better solution when there does in fact exist a better solution and then to spend an inordinate amount of resources solving a non-problem.

> Thankfully, you don't need to reinvent the wheel, the problem of cross platform multiprocessing in C has been solved at least a dozen times by now. It should be trivial to use one of these many options (or even write your own, even if you don't like the idea of "reinventing the wheel" it's certainly not particularly difficult either and can likely be lighter than any other more general option).

This is basically the issue he's complaining about. The author wants to reuse a wheel, but reusing the wheel in C is difficult. What are your options?

* OpenMP: Great, now you get the fun of figuring out how to cross-platform(/compiler) enable openmp in your build system (that you have to maintain yourself because there is no standard build system. Of course, you could also go with cmake or autotools to maintain the build system for you, but now you have similar issues with meta-build system stuff. Yay platform diversity!)

* Okay, let's use another library instead. Now you just have to have the user tell you where it's located. And maybe have to provide lengthy instructions on how to find it. Or you can try having the build system automatically download it on the fly. There's a lot of potential error messages you have to deal with, just to get at what ought to be a core feature of any modern programming language.

* Maybe you might want to vendor that library in your source tree, just to be sure you always have it. Now you get the fun of having your build system invoke another build system as a subproject.

* Screw other dependencies, let's just use a newer C standard version, since it is in C11. How do you get your C compiler to compile for C11? Oh wait, that's basically the same world of pain as it is to try to use OpenMP.

> Hardly any more dreadful than rust's solution to the problem.

Contrast this with Rust's solution.

* Your build system is cargo and you specify things primarily with Cargo.toml. This is as little work as the happiest happy paths of C/C++ build systems. And note you don't really choose build system; there is one ecosystem-approved build system--for C/C++, you have at the very least hand-rolled makefiles, IDE solutions, cmake, and autotools as major candidates, and many major projects very happily choose "none of the above" (with all the pain that entails).

* If you want to add a dependency in cargo, you add a line in Cargo.toml. That's it; you're done. For the aforementioned C/C++ build systems, even the happiest happy paths aren't that happy.

This isn't to say that Rust's solution is perfect (it has a fair amount of warts), but to assert that Rust is no better than C is an indication to me that you've never really had to actually support the build system of a project before.

> that you have to maintain yourself because there is no standard build system.

Since rust has no standard, how can it have a standard build system?

That aside, the fact that C has no enforced/bundled/included/coupled build system is personally a major win over rust. Why would you want to be forced to do things the way cargo does it? Not everyone has the same requirements. It's also a serious misconception that cross-platform building in C is is difficult, it's not as long as simply consider it from the beginning and test it (and ignore microsoft compilers and environments since it just much easier to cross compile for windows or compile via some mingw environment).

>Now you just have to have the user tell you where it's located.

No? You use pkg-config with the fallback option of having the user use PKG_CONFIG_PATH to specify a custom location for .pc files. If the library sucks and doesn't support pkg-config, either pick a better library or add an option to fall back if -llibrary doesn't find it. Realistically users should be encouraged to use their system package manager rather than just installing libraries all over the place. I agree that this could be made smoother but at the end of the day, this has never really been an issue for projects.

>There's a lot of potential error messages you have to deal with, just to get at what ought to be a core feature of any modern programming language.

I highly disagree that programming languages should have anything to do with package management. Feel free to make a programming language, feel free to bundle with it a decoupled optional build system, and feel free to attach to that an optional and decoupled package manager (see python) but to bundle them all together is incredibly inflexible and discourages me from using that particular language.

> Maybe you might want to vendor that library in your source tree, just to be sure you always have it.

Unless this is a commercial proprietary internal project, you NEVER want this.

> Now you get the fun of having your build system invoke another build system as a subproject.

If you still DO want it, it's usually trivial to do this. Linux distributions usually publish files which automate building all packages, these files usually don't span thousands of lines. If archlinux can do this with some shell scripts then it's not that hard to do it from any build system. That being said, you DON'T want to do this, it's just the wrong way to think about the responsibilities. You document the versions of libraries that your project supports, optionally you document that your project uses pkg-config or whatnot as a way to specify unconventional locations for dependencies. The people compiling your code follow these basic instructions and they get your code built. It's actually really not that hard.

> * Screw other dependencies, let's just use a newer C standard version, since it is in C11. How do you get your C compiler to compile for C11? Oh wait, that's basically the same world of pain as it is to try to use OpenMP.

C11 optional supports a thread model which was never thought about. So it's not an option here. That being said, you are seriously misinformed if you think that compiling against a specific C standard version is particularly hard. It's nontrivial to get compilers to completely drop their own extensions, but as long as you make sure your code is compliant with a specific standard and explicitly make that clear to the compiler's you're invoking, it's really not difficult. Certainly not as difficult as making sure that the OpenMP version is correct.

To be fair, given this completely incendiary article I would doubt that getting OpenMP to work is as hard as the article makes it sound, I've certainly never had these issues with it, but at the same time I don't have the time to look at the particular problems the project had so I will have to take it on faith that at least on that one particular point there might be an issue.

> * Your build system is cargo and you specify things primarily with Cargo.toml. This is as little work as the happiest happy paths of C/C++ build systems. And note you don't really choose build system; there is one ecosystem-approved build system--for C/C++, you have at the very least hand-rolled makefiles, IDE solutions, cmake, and autotools as major candidates, and many major projects very happily choose "none of the above" (with all the pain that entails).

I once again don't understand why you think having fewer choices is better. In my build system adding a new dependency is as easy as specifying the library and version number in one file. It can be like that if you want. The fact that C doesn't couple the build system to the language means that I can pick any tool I feel like being appropriate for a project and use it. It might be nice if there was some default build tool for C which could be used for the most basic projects, but I would be severely unhappy if it was as thoroughly coupled to package management and the language as cargo is. As it stands, rust is basically unusable without cargo, that doesn't really make me want to use it as a language.

I also don't know of any projects which chose "none of the above". Unless you mean projects which use meson but I don't think you mean that since it just looks like you simply missed it off your list.

> * If you want to add a dependency in cargo, you add a line in Cargo.toml. That's it; you're done. For the aforementioned C/C++ build systems, even the happiest happy paths aren't that happy.

How do I depend on a specific C library from a cargo.toml? How do I declare a runtime dependency on grep or ps or whatever. How do I declare a compile-time dependency on python (because for example I want to run python to generate some code which gets built)? The fact that cargo.toml solves a tiny fraction of problems (at great cost) while ignoring the rest doesn't really make me particularly excited about it. In C I can pick or design a build system which suits the task at hand, if I need to depend on python, I can document it (specify it has to be in the PATH) and run it during the build. I can make this as flexible or inflexible as I need. I highly doubt that it's possible to pick any random possible executable and add in one line the information needed to document the dependency on that executable, use it in the right way and integrate that into the build process. cargo is inflexible, I agree that it's nice if your project is simple, the fact that it's so inherently tied to rust means that it's annoying at best if it is not adequate for the task at hand.

>This isn't to say that Rust's solution is perfect (it has a fair amount of warts), but to assert that Rust is no better than C is an indication to me that you've never really had to actually support the build system of a project before.

Rust's solution is adequate for a small subset of problems not completely overlapping with problems I like to solve. It is inadequate for what I want to do, aside from the number of other issues I have with rust, it's just not as flexible as what you can do with C. I have had to actually support the build system of a project before.

> That aside, the fact that C has no enforced/bundled/included/coupled build system is personally a major win over rust. Why would you want to be forced to do things the way cargo does it? Not everyone has the same requirements.

There is a certain amount of irony in asserting the superiority of not having a standard build system when you immediately go on to advocate simply not supporting anything that isn't compatible with your way of building things.

I don't see the irony in what I said. Do you want to address what I said with a bit more nuance and detail than just "haha irony"? Especially given that you seeing irony in something I said doesn't automatically dismiss anything else I've said.

As a side note. Restricting yourself to rust and cargo, you're going far further than just discarding MSVC as a worthwhile target.

> Why would you want to be forced to do things the way cargo does it?

Because then you don't actually have to do anything at all. All the work is pushed onto the provider of the library. And they have less work too because 1. All their dependencies will "just work". 2. They only have to support one build system too!

There is ongoing work to integrate Cargo with build systems like Bazel, Buck, Meson, etc. Which is currently a weakness of Rust if you need to use those tools. However, note that this work is going into the tools themselves rather than every package author having to repeat the same work. That seems like a big win to me.

>Because then you don't actually have to do anything at all.

It's wishful thinking to suggest that all build related tasks can and will be doable with cargo. If cargo does not do exactly what I want, either it's impossible to do with cargo or is made more difficult by forcing me to use it.

>All the work is pushed onto the provider of the library.

What work is there to do for me as someone who depends on a C library? Ignoring weird edge cases like OpenMP, it's trivial to write in my documentation "Depends on SDL v2" and add 'sdl2' to my libs array or whatever particular build system detail there is.

> What work is there to do for me as someone who depends on a C library? Ignoring weird edge cases like OpenMP, it's trivial to write in my documentation "Depends on SDL v2" and add 'sdl2' to my libs array or whatever particular build system detail there is.

As someone who isn't overly familiar with the C ecosystem: suppose I want to depend on SDL2, and I want my code to work across windows, macOS and linux. What does a minimal build config look like (in a build system of your choice)? And how (as a second developer or on a second machine) would I then use that config assuming it was committed as part of the codebase?

GNUmakefile:

    LDLIBS += $(shell pkg-config --libs sdl2)
    CFLAGS += $(shell pkg-config --cflags sdl2)
    
    prog: prog.o

Then you just run `make` or `gmake` (the developer should be aware of what his platform of choice prefers to call gnu make) and prog.c gets compiled to prog using the appropriate flags. I missed out some important details like setting the standard revision but I wanted to keep this short. This will work with gnu make, gcc and clang on linux, mac, any BSD, and windows (under wsl or msys or cygwin). Cross compilation is possible too. Personally I would use redo as the build system but this is shorter in make for such a simple project. More complex projects can be made to work with this basic pattern. When using autotools, cmake or meson you also have plenty of good options although I'm not entirely familiar with them as I don't use autotools, cmake or meson.

Obviously this won't work if your machine is broken or if you don't have SDL2 installed in the correct place, but these are concerns for the package manager. The documentation simply needs to instruct you to install SDL2 (with development headers and the pkg-config definition). If you need a more advanced use-case such as you're trying to vendor the dependencies, you need to deal with PKG_CONFIG_PATH but I should stress that vendoring is NOT a normal use-case in the open-source-project-relying-on-sdl2 world.

The main deficit here is package manager. People are not writing yet another cross-platform package managers for no reason. Providing a consistent cross-platform experience is a value add. With "use system package manager", you'd need something like the following at least:

On Windows/MSYS2, run pacman -S mingw-w64-SDL2. On macOS/Homebrew, run brew install sdl2. On Linux/DEB, run apt install libsdl2-dev. On Linux/RPM, run dnf install SDL2-devel.

Which is no fun, not to speak of problems when versions in system package managers are different.

Using the system package manager means that your software can actually in the future be packaged for a distribution. This is a major win in the sense that if your software ever achieves any level of popularity, it is more likely to actually get packaged and as a result, your end users are more likely to actually be able to easily make use of it.

Distro package managers are a far superior solution than having 5 different language package managers you have to keep in sync and then having to compile your own software.

As I think this is a genuine question, I will answer accordingly. My build system of choice is CMake and my package system of choice is vcpkg. I will assume both are already setup. They are easy to setup on all of Windows/macOS/Linux by following the instruction. I tested.

First, you need to find out the package name. Run vcpkg search sdl2. It tells you that the package is called sdl2. It is trivial in this case, but if you wanted SDL 1.x, it could have been called sdl, or sdl1. vcpkg search will tell you it's sdl1.

Next, you install it by running vcpkg install sdl2. This will build from source. >1000 packages in the official repository should build cleanly, warning free, etc.

Installation tells you how you should use the package with CMake. It is two lines. find_package(SDL2), and target_link_libraries(main SDL2::SDL2).

To use CMake with vcpkg, copy and paste two lines to your CMakeLists.txt, and run CMake with CMAKE_TOOLCHAIN_FILE set to vcpkg.cmake.

Yes, C situation is bad, but it is not as bad as people think.

Rust has de-facto standards/monoculture. Things don't have to be stamped by ISO for everyone to agree on them. There is bare `rustc` for cases Cargo can't handle, but Cargo is good enough for almost everyone. People forgive (and hack around) some of Cargo's quirks and limitations, because there is a huge value in consistency and interoperability that a single unified build system brings. https://docs.rs can build docs for literally every Rust package. The Rust team runs unit tests of all public Rust packages when testing compiler changes. 3rd party tools integrate smoothly, e.g. cargo-deb makes a .deb from any Rust project, with no config at all.

> C11 optional supports a thread model

I've been fighting for C99 support 15 years after its publication, so a merely decade-old C11 seems too new and radical. MSVC of course doesn't support C11 threads.

> How do I depend on a specific C library from a cargo.toml?

It is easier if you rewrite it in Rust ;)

But seriously, you make (or usually use an existing) Cargo package for it. Cargo calls them "-sys crates". They contain an arbitrary build script that figures out what combination of pkg-config, vcpkg, brew, or vendoring is the right one for a given snowflake C library. This may seem like overkill, but it really is helpful and often necessary if you target more than a Linux distro. pkg-config is utterly useless on Windows. Gives out brittle library paths on macOS. Non-existent on iOS. On every OS it gives invalid results for cross-compilation, unless user works around that in a platform-specific way. Some packages like libpng, llvm or macOS openssl may have non-working or incorrect pkg-config files on purpose.

Would your build script include workarounds for Homebrew? Emscripten? If you're not a user of these, would you even know you need them? In Cargo people collaborate and share their build script workarounds as dependencies too. It does look like duplicating work done by Linux distros, but it's done for all platforms, so you get the consistency you expect from a Linux distro everywhere. e.g. openssl-sys has this monster https://github.com/sfackler/rust-openssl/blob/master/openssl... or all the pains to configure ffmpeg build, e.g for cross-builds or static linking on macOS: https://github.com/meh/rust-ffmpeg-sys/blob/master/build.rs

De-facto standards that aren't written anywhere and change over time are not a very fun thing to work against.

gcc, clang, python and I imagine quite a few other programming languages are developed to this day without the ability to run unit tests against everyone's code and as far as I'm aware they're not breaking everyone's code on a regular basis so the point you make about this doesn't seem that important.

But most importantly, you can have package standards without the tight integration that cargo requires (see python setuptools and pip for an example of a decoupled package manager and a decoupled build system).

> I've been fighting for C99 support 15 years after its publication, so a merely decade-old C11 seems too new and radical. MSVC of course doesn't support C11 threads.

MSVC is not a viable C platform. C in MSVC is just an afterthought and you're at the whims of Microsoft as to whether they want to support all of C or not. Complaining that writing C code which also compiles against MSVC is hard is not productive when MSVC doesn't intend on actually supporting C further than what Microsoft feels they can absolutely get away with.

>It is easier if you rewrite it in Rust ;)

Yes, of course, let's rewrite everything in rust. Because rewriting software has zero chance of introducing bugs.

Moreover, let's rewrite our core libraries in rust so that they can get statically compiled into software in such a manner that if there's a security vulnerability (very likely given you've just rewritten it) you have to notify thousands of people to update their version pins and then fix all the resulting fallout (because everyone pins everything, nobody cares about strict backwards compatibility). This seems like hell compared to the current situation. Proponents of the highly restrictive rust/cargo model are always complaining about duplication of effort as an excuse for why any given package can have upwards of 100 dependencies, but they are perfectly content insisting that people should duplicate years of effort by rewriting everything in rust.

>pkg-config is utterly useless on Windows.

It's fine if you use msys or cygwin

>Gives out brittle library paths on macOS.

That sounds like a macOS bug.

>Non-existent on iOS.

People compile things on iOS?

>On every OS it gives invalid results for cross-compilation, unless user works around that in a platform-specific way.

There's work being put in place to make this better[0]. But I would say the current situation isn't that bad, it's just a matter of setting PKG_CONFIG_PATH. Most build systems for C can handle this properly (assuming they're correctly configured).

>Would your build script include workarounds for Homebrew?

What workarounds are required on my end?

>Emscripten?

I'm sorry but targeting javascript is not something I'm ever interested in. I am of the opinion that the less support there is for this the better.

>OpenSSL and ffmpeg.

The problem here is that I don't believe that it's worthwhile to statically compile against everything. On windows and other systems, there already exist perfectly normal ways of acquiring a copy of OpenSSL and ffmpeg. I don't need to compile either of these as part of my build process. If the end user wants to use a statically compiled version of these libraries, they're welcome to figure out how to do that on their system of choice.

In summary, I don't see how forcing everyone to statically compile everything and rewrite everything in rust is a major advantage over being able to chose where your libraries come from and chose to dynamically link against existing system libraries.

[0]: https://www.freedesktop.org/wiki/Software/pkg-config/CrossCo...

Reply on news.ycombinator.com