So only people capable of auditing source code and build scripts deserve to be able to trust software? There should never be any other way to offer trustability?
This isn't a knock on the value of package managers or maintainers. It's just an obvious step in better security. It seems silly to argue that integrity among package maintainers is the only safeguard we need. I personally like the little piece of plastic that my laptop has that slides across the built-in camera. It's not a software solution, or even an electronic safeguard. It's even better than the little DIP switch on my phone. I say, why not?
I don't know about "deserve" but it's true that we have the knowledge necessary to understand what a script or program is doing.
> There should never be any other way to offer trustability?
Of course not. Someone else can audit it for you. If you trust that person, then you also trust the software that they audited.
Linux distribution packagers are the simplest example I can think of. If something makes it into a Linux distribution like Debian, it's pretty trustworthy. That's a big reason why we users like that model. It's also why developers hate it.