Yes, I agree. But note that the same logic applies to human beings too. Just because people say they are using chain-of-thought or tools to come to a certain conclusion, doesn't necessarily mean they are. Philosophers have been grappling with this issue for many centuries :-)