This is a valid criticism.

Upstream being AOSP helps a lot.

you cannot be certain, but at least the code has the chance to be publicly scrutinized. This is not the case at all with google binaries, so you have a net, objective gain.