"Japan has a higher GDP per capita than Alabama" is fake news. It's also confidently repeated by most LLMs. https://twitter.com/MatthewJBar/status/1681554646664634368

My assertion is that things like this will happen whenever LLMs are tuned to match political beliefs.