I'll admit that I haven't even figured out what it means for anything to "know" something.
But everything I've learned about, witnessed, and experienced with ChatGPT is consistent with a far more mechanical process than I experience with myself, my friends, or my community. I find it easy to spot the wires above the marionette and find there workings to be pretty intuitive. I have a comfortable sense of which ones to tug to make it dance the way I want, and when it's producing something unusual I've found that it doesn't take a lot of work to come up with a consistent, mechanical theory of how.
With people, that sense of clarity and comprehensibility hasn't really become apparent despite many decades of engaging with a bunch of them. There seems to be something far more sophisticated going on under the hood there, and they consistently surprise me and confound me in ways that ChatGPT doesn't even approach.
Maybe some AI model will achieve the same thing someday, and maybe that day will be soon, but for now ChatGPT looks and acts exactly like what the associated research papers would suggest.
Same! That's the overhanging concern I have when hearing comparisons made between artificial and human intelligence. I don't feel that human knowledge/intelligence is well defined enough to the point where we can meaningfully compare it to alternative systems.
But my gut instinct is that our neural architecture is composed of mechanistic biological compounds that gradually evolved sufficiently complex levels of organization to achieve our current level of cognition. Capacity for knowledge/intelligence/consciousness did not suddenly appear at some instant in our evolutionary history, rather it gradually emerged from increasing levels of neural complexity. I would rationalize the development of AI capacity for cognition in a similar light, that it will gradually emerge from increasing levels of complexity.
It's just so good at, frankly, at being a troll (keeping you talking while not caring about truth or being deceitful), that it has learned a bunch of truths, with emphasis on truths that keep people hooked on it's conversations.
I think that a core part of what it means to "know" something, is also an understanding of "not knowing". If you provide ChatGPT with insufficient data, it doesn't ask you to clarify anything to fill in the dots or try to eliminate a possible misunderstanding. It just, as you nicely put it, completes the sentence as best it can.
Ask a curious human a logical question or puzzle, and they'll often reply with a bunch of follow ups to fix what they assume to be a lack of knowing.
It's really good at faking logical reasoning abilities up to a certain point but it very quickly falls apart and starts returning nonsense.
I should be able to query specific details about a topic and get a consistent response with respect to its own and my own answers complete with citations of its sources. When I lie to it, it should know I'm lying. When I pose an argument it should actually debate instead of giving canned answers. When I ask something unfalsifiable it should recognize it as such instead of hallucinating. When I ask something in a deliberately vague way it should be able to pinpoint exactly what I made vague and ask for me to clarify that part explicitly. When I contradict myself it should ask me questions and continue the conversation instead of me doing all the heavy lifting.
Addressing the philosophy a little, it's pretty damning when the AI fails at extremely basic logical tasks. If even logical primitives are broken, how could it possibly prove to itself, much less to us, that its more complicated ideas are true? That still sidesteps what it means to "know" something, but I think it's a meaningful prerequisite.