After swapping an and b in your example it correctly detects the divide-by-zero, impressive.

After replacing the zero with an expression that must be mathematically evaluated to get zero, it still detects the divide-by-zero - that’s downright spooky.

https://whatdoesthiscodedo.com/g/6a8f359

ChatGPT’s inability to verify its claims limit its usefulness - but the example above is unsettling because it certainly seems to be evaluating things somehow, so what’s stopping OpenAI from enabling it to verify certain things through evaluation?