You're making the assumption that GPT doesn't automatically build a phonological theory of the english language from rhymes and lyrics and linguistic texts that are fed into it.
I just asked ChatGPT: Do "not" and "knot" sound similar?
> "Not" and "knot" do not sound similar. They are two distinct words that are pronounced differently and have different meanings. "Not" is a negation word that is pronounced with a short, sharp "n" sound followed by a long "o" sound and a "t" sound. "Knot" is a noun that refers to a tangled mass of something, such as rope or thread, and is pronounced with a short "k" sound followed by a long "o" sound and a "t" sound. Despite their similar spellings, "not" and "knot" are not homophones, which are words that are pronounced the same but have different meanings.
> Yes, the words "not" and "knot" do sound similar when spoken aloud. They are pronounced the same way, with a short "o" sound and a long "t" sound. However, they have different meanings and spellings. "Not" is a negation or denial, while "knot" is a fastening made by looping a piece of cord, rope, or other material around itself or through itself and then pulling tight.
I tried a few times and it gave me other answers, including something like your answer and another claiming they were similar but not quite the same.
> Yes, "knot" and "not" are homophones. Homophones are words that are pronounced the same but have different meanings and spellings.
My immediate follow-up: "Do 'knot' and 'not' sound exactly the same?"
> No, "knot" and "not" do not sound exactly the same, although they are pronounced similarly. [...] "Knot" is pronounced with a short, sharp "k" sound at the beginning
I've spend a long time testing this in ChatGPT, and no matter what I do it still gives results like this (paraphrasing here because it's down right now):
>What words rhyme with coffee? > doff happy toffee snuff duff
> Does "snuff" rhyme with "coffee"? >
Yes because they both share the 'o' vowel sound.