It caught me out with specific examples that violated my conjectures. In one case the conjecture held for all but one case, another conjecture was generally true but not for 2 and 3.
In one case it thought a conjecture I made was wrong, and I had to push it to think through why it thought it was wrong until it realized the conjecture was right. As soon as it had its epiphany, it corrected all its logic around that concept.
It was very simple stuff, but an interesting exercise.
The part I enjoyed the most was seeing GPT-4's understanding move and change as we pushed back on each other's views. You miss out on that impressive aspect of GPT-4 in simpler sessions.
I am not criticizing, but curious about your opinion.
I did similar testing of GPT4, and my observation is that it starts failing after 3-4 levels of reasoning depth.
That sounds more like an implementation or resource limitation, rather than an inherent limitation of the technique in general.
LLMs got lots of investments: 10s of billions of dollars and tons of compute, maybe more than any other tech in history, and can't crack 3 steps reasoning. It sounds like tech limitation..