The challenge here is the broader challenge with AI (eg self driving cars). Folks want the benefits but while also absolving themselves of the responsibility, liability and accountability for the quality of the output.

That’s a fundamental issue with so many of the proposed use cases for these things. You can’t throw an AI in jail for filing a false police report. Yes you can simply say officer must review but we all know folks are going to cut corners. Until someone goes to jail for filing a false police report and fails with a “but the AI did it” defense, I’d expect full blown shenanigans here.

We got a preview of this with the lawyer that filed an error ridden court filing and the judge threw the book at the author attorney and wasn’t having any of his “but the AI wrote it” defense.