Jepsen: ParticleDB 1.1.4
jepsen.io
jepsen.io
> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
Making it distributed opens a whole can of worms, which is where Jepsen shines.
It makes mongodb from the report a decade ago look good.
You can't be sure until they test it again
I mean did you see the "primary key equality" thing? Holy shit.
Please try to resist the urge of being a meat proxy.
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
What a waste of CO₂.
And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).
What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.
That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.
https://web.archive.org/web/20260518201315/https://particled...
https://web.archive.org/web/20260518200305/https://particled...
No. Wrong. False. All performance numbers are plainly invalid. As my coworker says, performance numbers don't count if the code is broken.
... executing SELECT * FROM releases WHERE 4 = 5 -- it's not just a slopdb, but TLA+ load bearing and 100% Jepsen ready
ParticleDB 1.1.4 ParticleDB 1.1.18.11.rc-22
2 rows returned
Sure, it still has some bugs resulting in occasional data loss, but it is pre-release so that's okay. Look at the speed, people!
I'm genuinely not sure who comes out looking worse - the coders for their reckless amateurishness, or the VCs for providing a misleading veneer of professionalism.
Edit: I like this 10 years old blog post in regards to explaining the problem much better than I could do: https://pron.github.io/posts/correctness-and-complexity
Yes. The combinatorial explosion is a great assessment of the "why".
But the problem is more nuanced than that because one has to assume that LLMs can reason accurately all the time which is demonstrably false.
The current LLMs cannot even state true sentences as true if they sound confident, without hedging.
Future LLMs may mitigate this, though it looks unlikely.
darkwi11ow, in the sibling comment, gave a substantive response on the underlying mechanics with a link. Godel's incompleteness theorems also yield similar insight. But I want to shine some light on a different phenomenon.
What happens in the future is up for debate, but the current architectural reasoning capabilities of LLMs are pathetic.
Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
Or try any novel architectural discussion and see for yourselves.
Modern LLMs cannot even confidently state many true sentences as "maximally" true as possible without hedging and tone policing as if we were doing politics ;).
Hence an expert is required at every turn.
Alas! It turns out it's really hard to do it right.
Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.
No one in their right mind will use this product ever.
Why would anyone believe you when you say this? From the report:
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."
Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.
> we're working on benchmarking now as well.
But you already said you were 1000x faster than Postgres??
How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?
"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.
Looking at some of the findings because I don't even have the time to pretend to read half of the failures leads me to believe that promoting this "database" should be illegal.
Go figure one of the founders is on openais board. Yikes...
> does not lead to high quality software.
Some software is correct by construction, in the sense that the overall design makes sense and the cases that don’t work are generally local bugs.
Some software is incorrect by construction, and it tends to work by some combination of luck and by carefully working around the incorrect design in most or all code paths that could expose the incorrectness. (AI can be quite effective at finding the relevant places to insert workarounds.)
ParticleDB sounds like the latter category. Plenty of large projects have portions that are in one category and portions that are in another.
As an aside, for the last day or two, I’ve been contemplating that the Linux kernel’s handling of folio pinning seems to be in the incorrect-by-construction category. Good times. No, I haven’t found a hard-to-fix bug that I would credibly characterize as a security or likely accidental corruption issue [0], but this seems mostly like luck.
[0] I do appear to have re-discovered a bug that some massive Anthropic dump from a couple months ago also found, and it’s getting fixed. If your application does “fixed” io_uring I/O or certain types of iommu-based or maybe RDMA on a memfd, you might plausibly encounter it on rare occasions. I didn’t bother making a reproducer.
I guess that's what happens when you vibecode a database...
root=> CREATE TABLE t
(a int primary key, b int);
root=> INSERT INTO t VALUES (4, 5), (5, 6);
root=> SELECT * FROM t WHERE b = 4;
a | b
---+---
4 | 5Yes, this is worse, but in both case the root cause was human carelessness, not AI.
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
In this theory, without evidence just like their db claims, they were looking for external evidence to shut it all down or force replacement of key personnel.. And Aphyr understood the assignment.
Perhaps they meant formerly, probably just a speech to text problem
(Edit for clarity: this is quoted from the linked article)
You could write predicates over non-existent columns:
root=> SELECT * FROM t
WHERE definitely_not_a_column = 4;
a | b
---+----
4 | 5
(1 row)
Or ask if 4 = 5
root=> SELECT * FROM t WHERE 4 = 5;
a | b
---+----
5 | 6
(1 row)
It must have been darkly amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so-slowly start to realize their software quality is not quite what their glazing agents had been telling them and that what they have on their hands is basically sub-mongo-quality slop. What a world we live in, huh?
I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL. Your example queries remind me of that.
[1] https://medium.com/dailyjs/the-why-behind-the-wat-an-explana...
> WHERE column = NULL and then query WHERE column <> NULL
NULL means what it means and is conceptually foundational to SQL. Comparing it to anything returns UNKNOWN, the “third value” besides true and false. And WHERE needs a TRUE predicate. So your implied fault that they “have no problem“ really is just a matter of you not understanding the tool you are complaining about.
To match against the NULLness of a value you must use special purpose operators. In this case
WHERE column IS NULL OR column IS NOT NULL You could write predicates over non-existent columns:
root=> SELECT * FROM t
WHERE definitely_not_a_column = 4;
SELECT * FROM t WHERE 4 = 5;
a | b
---+----
5 | 6
(1 row)
System.out.println(((Integer)4).compareTo((Integer)5));
> -1
So it's SELECT FROM t WHERE true;
Now, let's get busy blaming Jepsen for using the tool wrong.
No, therefore the condition is unsatisfiable and equivalent to WHERE FALSE. And that's letting you ignore the SQL spec, which says the query is invalid and should return "Column not found" because the column name cannot be resolved.
> We're comparing 4 and 5. Precedent?
I really hope you're trolling, for your own sake.
The condition is unsatisfiable, and the engine went ahead and gave a 0-warning result?
1) It would be no more fucked up than anything else that has been grandfathered into our tools.
2) If it already had mass adoption in the market, people would jump onto this thread to explain to me that I just don't understand the brilliant semantics behind language of why 4 = 5.
That's why I preplied:
> if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork
If 4=5 made it into the spec, would it be no longer up for criticism?
More cynically, I suspect that many low-quality database projects succeed or fail based more on marketing and hype than robustness. A dishonest product could simply take the line "engineers at Orkes were able to produce a highly sophisticated database system" from the report out of context, quote it on their homepage attributed to aphyr/Kyle/Jepsen, and reap short-term rewards and popularity. I hope I'm wrong.
And bugs like “WHERE 4 = 5”, or selecting from nonexistent columns, or selects permanently modifying table behavior, or any of a dozen others that Jepsen found … didn’t happen in released databases. Even young ones.
Prior to the age of LLMs, that level of bugginess wasn’t “everything has growing pains”. That’s the kind of thing previously associated with poor undergraduate CS homework or “the CEO’s kid wrote our database” corporate horrorshows.
> with no QA
That’s not immaturity. That is a choice. A “the car we are selling does not have airbags, seat belts, or brakes” kind of choice.
> Maybe you don't know too much about engineering
Save it. You have no idea what you are talking about.
Not even the grungiest version of MySQL 3.23 running on some shared hosting provider in the early 2000s would behave as immediately illogically as the database under test here.
Someone with your username should know this.
Postgres was already mature by the early 2000s - it was started in 1986 by a veteran who had written Postgres' spiritual predecessor in the 70's (Ingress). So no, Postgres didn't have amateur-hour data loss bugs in the early 2000s. What Postgres was missing out of the box were some features, like replication.
Well not until you'd been running it for a while and it ran out of tuples at least.
The ParticleDB devs definitely don't
Edit: I know @dang does nothing but maybe make it more likely he sees it.
writing to hn@ycombinator.com from your account associated email address.
This is such a huge reputational hit, seems like a terrible outcome for a Series B company funded by big name VCs
What does this say about their main product?
I can see it happening if they genuinely had no idea of how defective / non-existent their internal QA & testing processes were.
If anyone had any fucking clue they never would have engaged with Jepsen with the software in this state.
This company and everyone involved are going to have to legally change their names to have any shot at future trust.
FINALLY we know what happened with eMudhra's keys!
(Sorry. Really couldn't resist.)
By comparison, this report paints an absolute clown show. The amount of anomalies is perhaps understandable (a single architectural flaw can have a lot of impact) but the fact that they found so many disparate issues is telling.
The fact that they then said they addressed the issues, but didn't, is about the biggest red flag imaginable. I would never let any client use this software under any circumstances.
Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.
The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.
- "This is probably the most sophisticated spam attempt in ClickBench so far"
- "Unless someone tells me otherwise, I assume ParticleDB is fake."
"Written in Rust · Formally Verified · PostgreSQL Compatible"
In other words, if I end up putting up such a piece of software for a supposedly production-ready database, I deserve to be ripped apart by Jepsen and others. Somebody has to assert software quality; if it's not asserted internally, it must be asserted externally.
In yet another words, software development is a trust-based business. If you break it, such as by outright lying to people, then you deal with the consequences.
I agree with GP, that vendors should be somehow required or very strongly encouraged to participate in such audits. Not because they want to (as you say, most wouldn't like their dirty laundry aired thus), but because consumers deserve to know. U.S. government software groups don't like FISMA/RMF audits either, but they have to do them anyway.
Honestly? At some point there's a fiduciary-like obligation for it. What ParticleDB delivered is so far past good or bad, it's just straight ugly and then some. It's so poor that, if Jepsen couched it in any way, they'd be implicitly misleading their readers.
it takes a lot of of human hubris and incompetence to think you can herd AI agents to squirt out a better version of a multi-decade project in weeks, without a single human in the loop. I can't imagine the levels of hubris in prompting "Make a faster Postgres in rust. Make no mistakes" and immediately uploading the result for Jepsen testing without bothering to manually kick the tires a bit to see if your DB is fit for purpose.
I see a direct line between this attitude and the Proton CEO deriding "loser thinking" in a subordinate that dares suggest the AI output have some oversight before going to prod; there exist people who are so AI-pilled they get lost in the sauce.
SDKs are MIT license CLI tools and Studio will be also be MIT License
Server is not open source for now but free for developers
Afraid to show the slop?
e.g. https://jepsen.io/blog.rss returns a text/plain respond containing only "Nothing to see here; move along.".
Well done. Nice D&G riffing in the endnotes too
yeah, provided:
1. test cases are being designed to cover functionality
2. test cases are implemented correctly - i.e. they actually fail if the implementation is defective
3. the tests aren't all commented out
4. the test suite fails if a test fails
5. CI actually detects test suite failure
6. failing tests block changes from being merged (apart from initial triage to stabilise the suite)
Five years ago, outsourcing your software development (including QA) to the lowest bidder might have gotten you results like this -- if you didn't explicitly require automated tests as a deliverable, you wouldn't get any, or if you got some they'd be vacuous and not test anything.
EDIT: It's up now
It takes gumption to build a new storage engine. I remember when etcd[0] first made it's way into @aphyr's cross-hairs. We took all of the input very seriously because we knew that a data store which is untrustworthy is useless.
Resolving these issues likely won't be easy, but if you can get to the point where you walk away from Jepsen unscathed the trust in your software will be greatly amplified.
0: https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul