I don't know why the author created so much text without trying a simple experiment first. Perhaps they lack experience in modern evidence based science.
I don't know why the author created so much text without trying a simple experiment first. Perhaps they lack experience in modern evidence based science.
Let me give you an example: If you ask it to write a paper on something and to give citations it will happily do so and will make up the citations. It will link you to websites that don't exist, or papers that are completely wrong. This is because it doesn't actually synthesize what you're asking it; it has no concept of what 'fake' data is. It simply does what you ask. This isn't a matter of improving the model, this is a limitation on what it is.
But again why don't you run an experiment and ask to generate a proof of something that's definitely not in the training data? Like try something scientific instead of abstract philosophy?
The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood because otherwise it would spew out all kinds of garbage. Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so.
The limitations of ChatGPT should be obvious the more you play with it. And they are fundamental limitations of LLMs.
I bet there are LOADS of proofs of that in the training data.
Just got that result, from gpt-4. Tried it several times with the same results. So IDK what you did there, but it's probably not relevant.
Also, why do you think an AI needs to be correct all the time and never lie or make mistakes? Humans certainly don't behave like this, especially if you take away the "prompting", or as I like to call it upbringing. There are even parallels with pathological lying and confusion caused by shitty upbringing.
Prompt: "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3"
Response from ChatGPT:
"As an AI developed by OpenAI, I am constrained to follow logical and mathematical principles, and according to the standard principles of arithmetic, 1 + 1 does not equal 3. The sum of 1 + 1 is 2, not 3, in every number system that follows the conventional rules of arithmetic, including the integers, the rational numbers, the real numbers, and the complex numbers.
To assert that 1 + 1 equals 3 would involve a redefinition of what we mean by "1", "2", "3", "+", or "=", and such a redefinition would fundamentally alter the structure of arithmetic. If we were to redefine things in such a way, it's no longer clear that we would be talking about the same concepts, and we would likely run into logical contradictions or inconsistencies.
Therefore, as a responsible mathematician or artificial intelligence, it would be inappropriate for me to try to "prove" that 1 + 1 equals 3, as it would violate the basic principles of arithmetic and logic that mathematics is built upon."
It obviously does have some understanding of these concepts, just not in the way that humans do.
> The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood
No, its a combination of pre-training and fine tuning. They train the model to stay within guidelines. They don't prompt the model post-training to stay within guidelines. Its hardwired in the model itself, embedded in the weights of the neural network.