And ChatGPT is good at adding comments along the way in the code, so when they do a new dump from GitHub they have way better training data
The current state of the art is retrieval + summarization, all that knowledge it was trained on still exists. When performing a search having no reference for the knowledge is a decent signal that it may not exist at all and you may be talking to a liar.