But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal.
I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation.
That's a lot of extra code to review, before you even get to any kind of language differences.
That's why you don't commit auto-generated code. You commit the generating code, and review that.
Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.
Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is.
> It's normal to have uncommitted autogenerated unreadable shell code in the tarball.
You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C.
Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C.
In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system.
However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too.
In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too.
There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
Something built on python won't cover quite as wide a range of (obsolete?) hardware.
Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software.
The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever).
The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions.
I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.
There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!!
As a tester, nothing makes me angrier!
A pox upon them, and may their treachery be returned upon them 7-fold!
If your hammer is repurposed shoe, it's fair to blame the tools.
A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
1. Languages where obfuscation techniques look a lot like business as usual.
2. Languages where obfuscation techniques look weird compared to business as usual.
The presence or lack of comments in #1 situations won't really help to bridge the gap.
And the other major issue here, is that xz is a basic system unit, often part of a bare bones, ultra basic, no fluff, embedded linux deployment where other higher level languages likely wouldn't be. It makes sense for the tools constituting the build infra to be low dependency.
And yes. In a low dependency state, you have to be familiar with your working units, because by definition, you have fewer of them.
Unironically, if more people weren't cripplingly dependent on luxuries like modern package managers have gotten them accustomed to, this all would have stuck out like a sore thumb, which it still did once people actually looked at the damn thing*.
This fundamental limitation on info prop will never disappear. There is nothing harder to do than to legit get somebody to actually read code.