It is true that many of the neurons don’t seem appropriate (e.g. mixing two largely unrelated concepts), but,
well, that particular issue isn’t a problem with the hypothetical I describe, because “this neuron represents ‘either a fire truck or a nebula’” is still something we can describe.
But, yes, I suppose it is possible that there could be important parts of how the network works that aren’t amenable to understanding what a local part is doing in order to achieve a task.
Perhaps details in how parts behave which seem meaningless individually, and only have a useful effect when many such parts are put together across many layers.
But, I still think that it is important to note that, when trained in a diverse way, representative of how the actual distribution, there are at least, many parts of the functioning which can be understood.
Trained NNs aren’t inherently completely inscrutable in how they accomplish things.
But yes, you do have a point that we probably can’t really rule out that there are important aspects of the functioning of these networks that can’t be understood locally in a human-meaningful way, even if many parts of it can be understood.