I mean, this is to be expected right, the LLMs are operating at a scale of billions (trillions) of parameters, so it's impossible to explain how every single step or response works.

Overall, people can grasp what an LLM does at a high level. But to debug exactly why an LLM gives a specific response to a specific query, one would have to debug hundreds of billions of parameters.