Yes, an ML model that infers B from A might not "understand" what A or B are....yet. But what is it to "understand" anyway? Just a more complex process in a different part of the machine.
If the human brain is just a REALLY large, trained, NN, there's no reason that we won't be able to replicate it given enough computing power.
I think one clear sign that the human mind is more than just a big NN is how large neural networks are already.
Take GPT-3, which is was trained on 45 terabytes of text and has 175 billion parameters. Contrast that with the human brain, which has around 86 billion neurons and is able to do much of what GPT-3 can do with only a tiny fraction of the training data. And it has to be said that while GPT-3 has more competency than an average human at some text generation related tasks, the average human brain is vastly more capable than GPT-3 at any non-text related task.
So for neural networks to approach human level capability we would need a whole stack of GPT3-ish size networks for all the other non-text related things the human brain can do: speech, vision, motor control, social interactions, and so on. By that point the amount of training data and parameters is so astronomical, there can be no question that the functioning of human brains must be significantly different than that of contemporary computer neural networks.
To be clear, I am also a materialist and subscribe to the computational theory of mind, but just based on the size of training data alone, it seems obvious that human brains work differently than neural networks.
If I read all day everyday from the moment I was born until now, I couldn't have read 45 terabytes of text.
I guess if we want to reduce the comparison to something vague like "the human brain and neural networks both developed over many iterations," I could agree with that, but that doesn't seem very interesting.
If the two were actually comparable, it would look more like: 1. Use NN (or GP) to develop set of hyper parameters 2. Give said HP to neural nets that, thanks to the HP, can be trained on a very limited said of data.
I am not aware of any successes with methods like these.