You're right in that a binary blob may just contain a backdoor and analyzing it is very hard.
There is Replicant ( http://www.replicant.us/ ), a version of Android that aims to get rid of all binary blobs. There is also CopperheadOS ( https://copperhead.co/android/ ).
Related reading: https://blog.torproject.org/blog/mission-improbable-hardenin...