I was playing a bit with ChatGPT and I wondered how much actual knowledge could be stored in that model. So I asked ChatGPT whether it knew the song by Franz Schubert called "Am Feierabend" and if so, if it could tell me the subject/meaning of the song. "Certainly I know this song", responded ChatGPT and then gave me -- with total confidence -- a completely wrong answer about the meaning of the song. In fact I was a bit baffled by the authority with which is spouted this nonsense answer. A truly intelligent system would be aware about the limits its knowledge, right?
So I think that with a minimum of creativity teachers can come up with questions that can easily stump ChatGPT.