Postgres 19 delayed after AI reviewers find more bugs than humans can fix
theregister.com
theregister.com
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
This is why code is going to be automated, and the software engineer is a link between organizational intent and the code. This is what I’m working on.
Finding the holes in your specs under automation: time spent is returned 10x
Writing code: Human 15 lines per minute. LLM: 600+ lines per minute
Functional Validation: same time per iteration. Fewer bugs with LLM when properly specified => fewer iterations. Increased awareness of LLM escaping specification boundaries feeds back into increasing quality of spec writing => fewer bugs.
So you get permission flags and multilayered data structured with complex interfaces, and there is never any sense of "Wait a minute - we don't need this."
It's much like the way AI art looks meaninglessly busy with too many random lines or other elements, and AI writing is always exactly, precisely, specifically, unnecessarily adverbial.
This happens if the model does not have good context regarding what is being built, why and for whom.
For one of my side projects, my CLAUDE.md has a line that says "the users of this product will be non-technical people who find technology scary." And you know what? Claude always steers me away from complexity when working on that project.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
How much of its judgment would be driven by the data vs. its own internal knowledge? How much might be driven by old vs. new discussions that might not represent new directions of the project? How much might be driven by over-matching against how bugs in less mission-critical code paths were handled?
There's no way to quantify the answers to these questions. Which is not to say this applies to every problem out there! But human judgment is still a bottleneck for design questions on widely, disparately-used complex systems.
Many of us have already experienced that uncomfortable revelation, where a leading-edge LLM actually comments correctly on our questionable judgement. You will, too, soon enough.
Help me out. What'd I miss?
In my case, rather than finding a different profession, I chose to walk through the open door to the next level of the same profession.
You seem more concerned with shining your AI badge than understanding what I wrote.
Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.
Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.
OK, so this is slightly different since it's more about how runtime behavior than design-time behavior, but... I'm in a SaaS workplace, and I've been trying to seed the following idea:
1. The very cheapest/fastest end of the market will be eaten up by Job Bob's Bait Shack stuff. Cheap wrappers around a prompt that goes "Pretend you're a computer that obeys the following rules" jank. A race to the bottom we can't win.
2. Most of our (desirable) customers will (eventually) realize they can't tolerate systems which casually or unexpectedly violate known business rules. That leads to angry consumers, junk data, broken workflows, and lawsuits.
3. When our product can capture and enforce those rules with confidence, that's both an important differentiator and also a kind of moat/stickiness.
4. Therefore there is a strategic business justification for Engineering to fence-off where we use LLMs in production. When we use them for design, it's to create better deterministic systems than before.
I'd say pre-LLM, just because I don't want to surrender the term "AI" to one particular subtype without a fight.
Some of those complex tools would be "Good Old-Fashioned AI", which generally means stuff doing symbolic logic rather than mystery-weights.
So… are we gonna use the extra productivity (and money) from coding agents to hire QA people? Oh boy, is that highly unlikely.
But I like your take. I wish it were so, perhaps we wouldn’t have had Tahoe.
I don't see any theoretical reason that this weakness must remain for longer than six months or so. I mean, both Claude and GPT are very good at doing code reviews and that's a form of QA.
None of that is because AIs aren't good. It's just plain information.
An AI could sit there and interrogate the user for this information. In that alone, it would already have exceeded what a lot of people are going to want from a system like this. People, normal people, want AIs to magic stuff into existence, not spend hours interrogating them for a design requirements elicitation. That's a barrier a lot of people will not cross, and the AI companies, who fully realize that, aren't building that way.
Normal people do not want to even do the work it takes to fully specify a problem, even if an AI walks them right through it, let alone the verification that the work is correct, and actually what they intended rather than what they asked for.
And of course, in practice, AIs are not emitting bug-free code, miss entire requirements at a shot, and have a variety of other characteristic failure modes in this space that I expect are fundamental to the LLM architecture and aren't ever going to leave. If they continue to be able to make bigger systems they will simply continue to make correspondingly big errors and omissions in the process. No comment as to what the next generation AI architecture will do, though.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.
There's a big difference between randos getting agents to hallucinate bug reports on GitHub, vs. when your own internal workflow of LLM-assisted code review is identifying real, legitimate problems in your new features.
This totally matches my experience. It's actually kind of insane how well LLM's like Claude are able to find edge cases, race conditions, overflows, states you didn't think could be reached. It really makes you think about code quality in a whole new way. Becoming aware of so many potential bugs when you write the code, vs. finding out from users over the coming months/years -- it forces you to really consider the costs of quality all at once in a way that simply wasn't possible before.
There’s plenty of ways in which you can guide LLMs, and sandbox them, to get a much higher hit rate.
And that’s before the increase of validity of bugs compared with a year ago — we’re past the peak of slopageddon, and maybe even at a point where a higher percentage of security bugs reported are valid than was the case pre-LLMs.
Either way, unfortunate but a very understandable decision.
If I had to root cause why we're finding so many bugs now, I would attribute it to the limited Postgres test suite. AI is merely uncovering bugs that already existed. For the most part, Postgres relies on individual regression tests to test each feature. While regression tests are great for making sure features don't break (yes, SELECT 'Hello, World' LIKE '%Hello%' still returns true), they actually aren't that great hitting the edge cases (passing a string that's over 2MB to a specific function will trigger a buffer overflow and crash your DB). The main way Postgres has prevented bugs in the past has been by minimizing risky changes and having lots of human review. As we're learning now, there were lots of gaps that were not caught by the existing process.
While the Postgres team is skeptical of AI, I've had lots of success in using AI to scale more advanced forms of testing such as Deterministic Simulation Testing, differential coverage guided fuzzing, and formal verification. To put it in perspective, in the process of doing testing for pgrust, I've uncovered ~50 different bugs in core Postgres. The bugs I'm finding aren't through AI review, but are instead through using AI to do more advanced testing than regression testing. All of the techniques I'm using existed existed pre-AI, but aren't used in meaningful capacity as part of the core Postgres development.
Right now I'm going all in on formal verification. I wrote a compiler to translate C and Rust functions to Lean and am then having an LLM write a proof that the two functions are identical. So far I've been able to verify 10.5% of the codebases behave identically (10.5% = 67k/640k LOC of Postgres). I'm finding details as minute as a parsing function where Postgres trimmed spaces using isspace() and pgrust only trimmed ascii whitespaces. These aren't cases you would find with traditional regression testing or human review. The thoroughness of the types of bugs I'm finding is giving me confidence that that this is catching more bugs than any sort of manual testing could. I don't currently see any reason that I won't be able to scale up the formal verification to cover 100% of the codebase. The only issue is it takes a long time to do.
[0] https://github.com/malisper/pgrust
[1] https://malisper.me/the-four-horsemen-behind-thousands-of-po...
You look at Postgres C functions and prove they are identical, but the license is AGPL without mentioning Postgres authors. Consider me confused.
We translated from C to Rust, rearchitected several components and are working on rearchitecting more. For instance, we replaced the process-per-connection model with a thread-per-connection model and added a columnar format + new query executor that's hundreds of times faster than Postgres for OLAP workloads.
> without mentioning Postgres authors
We have a NOTICE file which references the Postgres license here: https://github.com/malisper/pgrust/blob/main/NOTICE
As a lead contributor explains in the article, if they were to ship some of these broken features, they wouldn’t be able to fix them until v20, rather than patching them on-the-go. Sounds like a Very Bad Idea given many installations stay behind for some years.
Relational databases are one of those things where you really do want to get a whole lot closer to perfect than a lot of other software.
Not every time. But often enough.
At a micro scale their code is right, but tons of invented complexity no sane person would introduce. I've found the two most common triggers are the models polluting their own context windows (poorly worded instructions in comments/issues/AGENTS.md) and a poor "taste" for breaking changes (they'll constantly try to make their additions non-breaking, and if prompted they can make breaking changes, it has to be very carefully scoped or they start assigning great reward to ripping out things)
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
Just because *you* can avoid it doesn't mean it's not an issue with the tooling.
Also maybe I'm a curmudgeon but I generally at least mark-one-eyeball all the tests it's generating and tell it when it's being extra dumb.
Anecdata, but most of the time when something doesn't smell right to me looking at genAI code, the root cause is ridiculous tests that made their way into the repo. And then the agents built up around it.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
And as others have said some models people prefer for implementation work are less prone to this.. 5.6 Sol, 6.1 Sol, Opus 5.5
Just don’t post the unedited AI “bug report” as an issue please. A human should make sure it’s actually a real problem before dumping it on devs.
That's what I was thinking as well. Wouldn't the majority of the bugs not also have existed in Postgres 18? The version that everyone is using atm...
There is a similar thing with AI detecting vulnerabilities. It's not that the vulnerability isn't real, but just that in most cases for these vulnerabilities to exist there also needs to be a very specific set of circumstances as well making the likelihood of the vulnerability to exist in the wild quite low.
A common misunderstanding.
The # of bugs lurking in a codebase is finite. At some point, the only way to find more is introduce fresh bugs through new code.
What we're seeing now is a transition phase where pre-existing bugs are found (and hopefully fixed) at an increased rate.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
It's a endless money loop.
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
Naturally, EDB is an AI shop (https://www.enterprisedb.com):
"Agents need direct, governed access to data at the source. EDB Postgres AI is built for that."
Of course, nothing ever has worked in PostgreSQL before AI!
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.