I know this is still a different trust model, but, IMO, not too different.
I know reproducible builds are on the radar for Homebrew in the future, and that should ease some of your concerns too.
For a good thread on Homebrew security checkout https://twitter.com/c_pellegrino/status/1093195802871246848
It also means, though this is only a problem with third party channels, that it's possible to get into a situation where you can't even build from source at all. I was working with one of our custom packages, where for the life of me I could not figure out how to get the build to work. Turns out it had been hardcoded to run on a very specific machine, and assumed access to a very specific network file system, and I wasted about a week of my life trying to untangle those assumptions before giving up. If that software had been designed from the beginning with the idea that it ought to be possible to build from source from anywhere, we'd presumably be in a different situation now.