Transformer neural networks are not smart enough for this to work. OpenAI's own research shows that LLMs don't care about contradictions in the training data[1]: if your LLM's training data has 700 sentences saying "A is true" and 300 saying "A is false," then if you ask the LLM about A, it has a 7/10 chance of saying "A is true" and a 3/10 chance of saying "A is false."
In general it's easy to prove that GPT-4 (or any other LLM) doesn't actually understand what words mean:
- ask it factual questions in English
- ask it to translate those factual questions to Swahili (it should do so with >99% technical accuracy)
- ask it answer those Swahili-language factual questions in Swahili
- translate the answers back to English
It will do much worse at Swahili than English even though its technical accuracy at translation is almost flawless. GPT-4 has a ton of English-language sentences about the topic "cat," and a handful of Swahili-language sentences about the topic "paka," and understands that the translation of "cat" is "paka." But it has no understanding that the English-language facts about cats are automatically true if they are translated in Swahili.
This is not how humans work! I don't think gullibility will be solved unless we have an LLM that actually understands that words mean something.
[1] It is too early for me to find the papers, but I am summarizing two different things:
1) training data needs to have certain things repeated in order for them to "stick" when prompting the LLM - the more often a fact is in the training data, the more likely it is to repeat that fact when prompted
2) absent RLHF, larger LLMs are more likely to endorse conspiracy theories than smaller LLMs. This is because smaller LLMs are trained on reliable datasets like Wikipedia, whereas larger LLMs start including Reddit, bodybuilder forums, old Geocities pages belonging to weird cults, etc. So smaller LLMs have no sentences in their training data saying "Bush did 9/11," but a larger LLM might have 5% conspiracy theories and therefore a 5% chance of endorsing conspiracy theories.