>What's the difference between a model that predicts intelligent output, and intelligence itself?
It doesn't directly attack the question, but gives a neatly constrained thought experiment which you can see running.
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
The first time we both improvise and memorise. But we have no idea of the underlying complexity of the edge cases. This matches what the imitation game does.
Then we practice and practice, and we start to understand, to really understand what's going on. This is in parallel with learning to deal with the edge cases.
Then, many years later, we learn everything again, but based on an internal world model that can predict and explain just everything about what we are learning. And that's when we actually learn. After we already know it :)