> asking questions where it is easy to judge the factual accuracy of the responses seems a pretty reasonable test to be running.
It isn't reasonable if that isn't what the system was designed to do.
It would be a poor test of my general practitioner's competence to ask him calculus questions and conclude he doesn't know what he's talking about because he can't answer them.