Oddly, despite LLMs being these huge networks with billions of parameters, we still probably do understand it better than we do our own brains.
Human brains and cognition do not work like LLMs, but that aside that's irrelevant. Existing machines can explain what they did, that's why we built them. As Dijkstra points out in his essay on 'the foolishness of natural language programming', the entire point of programming is: (https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...)
"The virtue of formal texts is that their manipulations, in order to be legitimate, need to satisfy only a few simple rules; they are, when you come to think of it, an amazingly effective tool for ruling out all sorts of nonsense that, when we use our native tongues, are almost impossible to avoid."
So to 'program' in English, when you had an in comparison error free and unambiguous way to express yourself is like in his words 'avoiding math for the sake of clarity'.
"A short look at the history of mathematics shows how justified this challenge is. Greek mathematics got stuck because it remained a verbal, pictorial activity, Moslem "algebra", after a timid attempt at symbolism, died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism —a vain attempt at verbal precision!— thanks to the carefully, or at least consciously designed formal symbolisms that we owe to people like Vieta, Descartes, Leibniz, and (later) Boole."
LLMs are nothing else but the exact reversal of this. To go from the system of computation that Boole gave you to treating your computer like a genie you perform incantations on, it's literally sending you back to the medieval age.
Saying an explanation it and actually knowing why you did it are two different things. That's exactly my point.
And then Boole's quote -- good quote, but I think you (not Boole) are conflating precision with motivation.
Now, physics says that everything can be explained mathematically, including the human brain. Obviously, on some level, an LLM can be explained. But despite hundreds of years of science, we still don't understand the human brain. Some systems are just really complex and difficult to understand.
Given all of that, I see no reason to assume that we'll be able to understand LLMs anytime soon. Especially given we keep growing more complex ones.