Just an example try asking it to write Battleship in Python. It does very well. But then, try changing the rules a bit, and adding some requirements. It spits out code that looks correct if you squint, but has terrible style, and often doesn't even work. Check out the test prompt [1].
Maybe GPT-4 is 1000x better at coding, but barring that, I don't see ChatGPT being useful as a replacement for human coders.
[1] Prompt: Write a version of Battleship with modified rules for sinking a ship. In this version, a ship is sunk when the "head" and "tail" of the ship (its beginning and end parts) are hit. Hits to the center do not matter. Use object oriented programming, with classes for the Board and for Ships. There should be support for multiple ships and two board instances. Write it in Python.