Hacker News Reader

Comment by moffkalast | Hacker News Reader

Has anyone tried using an LLM as an LLM detector yet?

It works…ish. GPT4 is pretty good at detecting what it wrote but I was able to get a false positive with the United States constitution. Or maybe we can go deeper and say maybe it was AI generated?

So if you submitted the US Constitution word for word for your assignment you wouldn't expect the cheating detector to flag it? How isn't this flagrant plagiarism?

I would expect a plagiarism detector to flag it. I would not expect an "AI wrote this" to flag it, because it was not written by AI.

An AI detector is basically a plagiarism detector trained on AI datasets. Fundamentally it's about discovering writing that is too similar to existing writing, which is why I'm concerned about false positives.

Yeah, which is why "AI detection" is crap. It's basically the plagiarism detector with the false-positive dampening turned off. (I know, I know, it may not be exactly that under the hood, but I think that's what it amounts to in practice).

I think you and I agree. I'm just saying that the "AI wrote this" flag on something that's notoriously not AI-written should be enough to reject outright the use of these detectors.

Like a mechanic who warns me that my Tesla doesn't have a muffler. They're technically correct, but I wouldn't trust them to diagnose anything in the future.

That is the implementation that many of these approaches are doing.

It is a very, very hard problem.

It is an impossible problem. An llm can produce any combination of words. A human can produce any combination of words.

Reply on news.ycombinator.com