"So they can definitely tell you the combined liklihood that the answer is 'Peru is a cat' vs 'Peru is a country' and provide you the exact statistical likelihood of each..."

...in the context of the texts that the LLM is built on. Not in the context of the real world, where P('Peru is a country') = 1.0 and P('Peru is a cat') = #cats named Peru / #things in the world (or something).