build.rs is a source file that you can audit. A binary that has no reproducible build is not auditable even if anyone wanted to.

A single person does not audit all of their dependency tree, but many people do read the source code of some if not many of their dependencies, and as a community we can figure out when something is fishy, like in this case.

But when there are binaries involved, nobody can do anything.

This isn't the same as installing a signed binary from a linux package manager that has a checksum and a verified build system. It's a random binary blob someone made in a way that nobody else can check, and it's just "trust me bro there's nothing bad in it".