He's talking about the practical effects of fallible humans using the version of Bayes advocated in the original post (citing Yudkowsky and Muehlhauser).

Pulling numbers out of your backside and running them through a process makes you more confident than just pulling them out of your backside directly, but I've yet to see evidence it does more than increasing your confidence.

Oh yes, I wouldn't question the math -- it's well beyond my expertise, and I've heard of way too many implementations of the theory to refute it at a theoretical level.