That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.
You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one.
A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!
Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.