You can literally read their thoughts if you run an open model, they look like pretty human thoughts to me, albeit a neurotic human.
I can write a program to produce a string that looks like human thinking, is it human thinking? Of course it isn't. It's such a silly comparison.
Neural networks in machine learning/AI are comparable to neural networks in human brains. What made you think they aren't?