stay in the loop friends
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
The optimizations on PostgreSQL are real if done correctly, it's just a very big project.
My new definition for The Singularity is not about Ai intelligence, it is about humans handing over critical thinking regardless of Ai capabilities
Ai companions are where I put the highest p(doom)
the project has 3 mega commits of claude slop
will you be removing the 1600x claim?
Depends on who wrote the original code. With postgres its sus, but if it's another AI...I wouldn't be surprised. I also saw code writtem by humans that is a result of "I can't figure out how to make this safe and concurrent, so imma just place giant mutex here", and it like 100 lines change to fix that.
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
I use LLMs a lot, and since ~GPT-2 times LLMs are not doing anything as broken as `repeating the word "the" until it fills up the context`.
Still would never have predicted this. Even 2.5-flash was pretty good. Never saw anything like this happen.
things have definitely improved significantly, however "never say never" is still good advice
> It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
We can agree that the pgrust project messed up and should have been more careful but your blanket statement lacks support.
"loss of control" is a phrase being used for a reason, the agents do things we do not want them to, even in the context of instructions to the contrary. The current training regime has incentivized "complete the task any and every way you can" - the bot judgement is very poor across the board