I am surprised that you can even say "it's not very good." The sorts of prompts I'm imagining I would expect to trigger the "I'm afraid I can't do that Dave" safeguards every time. I guess yeah, I'm just imagining using it as an advanced fuzzer but I think the thing about using an LLM is you can take the fuzzer code and ask the LLM to just generate slight variations, feed that into a test and ask the LLM to flag ones that look like they might have been an exploit. And when it generates nonsense code, you just throw out those runs. But on the other hand hallucination feels like an advantage here since it's going to do things you never would have thought to test.