(Which is why prompt engineering is a thing! The art/science of phrasing the prompt so that the immediate blurted response is more likely to be correct.)
prompt-engineering to get the answer is analogous to prompting the fortune teller to tell your future - ie, you have an intended answer that you expect (or wish for in the case of a fortune teller), and you tweak the prompts to get there.
Well, you might not know the answer, even if you know some properties the answer must have and some processes that might lead to the answer.