While your application of the halting problem here is wrong you actually can use it to prove your point about chatGPT.
Have it recite the halting problem
In a new chat have it write a python program that tells you if a given string containing a python program will halt.
It's great at hacking, and completely incapable of engineering.