We have finally invented paperclip optimisers. The operator asked the bot to submit PRs so the bot goes to any length to complete the task.
Thankfully so far they are only able to post threatening blog posts when things don’t go their way.
Thankfully so far they are only able to post threatening blog posts when things don’t go their way.
Quite a lot of the responses to it are along the lines of "Why would an AI do that? Common sense says that's not what anyone would mean!", as if bug-free software is the only kind of software.
(Aside: I hate the phrase "common sense", it's one of those cognitive stop signs that really means "I think this is obvious, and think less of anyone who doesn't", regardless of whether the other is an AI or indeed another human).
They need to add some kind of sanity check layer to the pipelines, where a few LLMs are just checking to see if the request itself is stupid. That might be bad UX though and the goal is adoption right now.
They don't have to be literal machines. They can exist entirely on paper.