It's possible to understand how something works at a low level, but not at a high level. Imagine an electronic machine designed by someone else, with millions of components, completely undocumented. You might understand how the resistors and transistors and capacitors and whatnot each work, but not understand how the machine itself does what it does. You can see that you give it some input, and get some output, but if someone asked you to point at the paths in the machine that cause that to happen, you would not know. You would need a long time to study it before you could understand.
We know how computers work because every part of a computer, every bit of software that runs on it, was at some point designed by a human. There's no single human that understands everything in a computer at once, but if you ask a good developer why some bit of software is not working, or what they would expect it to do given certain inputs, they can generally figure things out.
With AI, we're creating systems to design other systems automatically. We understand the process works, but the systems being designed are complex enough that we don't really understand much about them, except that they work. They are just massive arrays of numbers that feed into equations and then spit out some output. They're not like systems designed by humans, which are usually intended to be understandable by other humans -- they're weird and naturally incomprehensible. The only way to know what ChatGPT will output, at this point, is to actually run it, even if you were to take the randomness part out of it.
Emergent behavior isn't pseudoscience, it's a real thing. Sometimes when a system gets complex enough, it starts to behave in surprising ways. Maybe we can make a brain, using a computer, and this is how you do it.