Hacker News Reader

Comment by the_duke | Hacker News Reader

Brutal!

> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion

> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested

View on HN

Fortunately the list of VCs funding this with millions and clearly zero due diligence is easy to find. I will add them to the address group for my next Nigerian Prince AI startup email distribution list.

They all got played by some ai dunning kreugers. The investors might not know it but anyone who knows anything about dbs knows that failing a jepsen test this badly is not the start of something. it's the end of something.

Looking at some of the findings because I don't even have the time to pretend to read half of the failures leads me to believe that promoting this "database" should be illegal.

Go figure one of the founders is on openais board. Yikes...

I guarantee you that every investor knows they are not knowledgeable about technical details. So, they have a person on call that they’ve designated as their technical advisor (or even entrepreneur-in-residence, depending on how much the investor is paying this person).

The problem is there’s no incentive for this advisor to admit they are out of their depth because the money will stop flowing: some flavor of “if you can’t help me on this, then what am I paying you for?”.

Experienced investors, a.k.a. those who have previously been burned and learned from it will have multiple technical advisors categorized by their past career specializations.

The other investors will continue to rely on the advice of one jack-of-all-trades with questionable results.

I've been that advisor. You'd be surprised how often investors are straight-up conned by founders into becoming hype men for broken-by-design businesses as soon as the advisors are out of the room.

That's not because investors are idiots. A little bit of it is because a certain amount of credulity/faith is a necessary skill in that business. But most of it is just because humans are fallible and trickable in general. Advice from an expert can only help with that so much.

I'm part of the ParticleDB team. We took these findings seriously and have spent the past few months addressing them and strengthening our testing. We now run sustained Jepsen workloads daily alongside our own test suites. We've also narrowed the supported transaction modes to focus development and testing on core correctness.

Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.

Fixing clearly identified issues is table stakes. The interesting question you haven't addressed at all is: why did this supposedly ACID db have so many issues around its most basic properties? Smells of a design issue.

Honestly best thing to do is to delete this and wipe out its existence and start a new project from scratch and rebuild the reputation for the company.

No one in their right mind will use this product ever.

Why did you pay Aphyr to find all these holes in ParticleDB? Didn't you know about them before? I wish you the best, but this doesn't sound reassuring.

Psychosis?

> We now run sustained Jepsen workloads daily alongside our own test suites.

Why would anyone believe you when you say this? From the report:

"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.

Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?

Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."

> I'm part of the ParticleDB team.

Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.

We have extensive unit tests and integration tests that we run in our build process as we test the db.

Show us.

both "extensive" and "test" are the load-bearing terms there

at this point even “run” is in question

What you're missing is credibility.

Hi Claude, can you fix all the Jepsen findings and push to prod. Thanks.

you joke but that sounds like how many of the fully AI-psychotic shops operate

There is no doubt in my mind this is exactly what they did after this was published.

I mean I assume that that's where, say, 3.3.2 came from; the database grows _new_ bizarre brokenness as stuff is fixed.

I'm genuinely kind of fascinated whether there was a _design_ here, or if someone just said "claude, postgres but faster pls". Some of the issues are truly incredible.

Where is the TLA+ proof?

TLA+ does not do what you think it does.

Go on, you're not saying anything yet.

I dont do consulting for free. I set you up already on the path of enlightenment :-)

If you took the findings seriously why didn’t you take the outrageous claims you were making in your own copy seriously? Why couldn’t you provide cogent analysis of these issues during the process? How did you hack up your db so bad in the first place?! This thing looks so bad bro I can’t help but think this kind of assertion only adds fuel to the fire.

> we're working on benchmarking now as well.

But you already said you were 1000x faster than Postgres??

I would highly recommend leaving the industry. You are not at the caliber required for this kind of work, and you are not aware that you are not at the caliber required for this kind of work. You bring shame to our profession.

I wish more of us had the courage to speak up and say things like that. thank you

I second this. I wish more people spoke up against gross negligence abd incompetence like this.

should you have done that in the first place? its a database engine

I struggle to imagine how an organization which is capable of taking things seriously would have produced software with these issues to begin with.

What kind of introspection did you do on the root cause of those issues? "Data from deleted table reappears in new vaguely-similarly-shaped table" or "simple row write just doesn't happen" are bugs that are quite impressive to accidentally happen!

How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?

"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.

Writing databases is an EXTREMELY fun learning endeavour but it is truly difficult to do correctly.

Once I started writing one (I could say I still am) because I was unhappy with most of them and also, to see how hard it could be

Alas! It turns out it's really hard to do it right.

"But AI can oneshot Redis, so we can just vibecode it."

"a library of confusion" is exactly what you should expect from a 100% vibecoded product as complex as a database.

I wonder if it’s vibe coded by amateurs

I would guess in a matter of 1 - 2 years, a random amateur vibe coding a brand new DB will get much better reliability and correctness guarantees than whatever this is. This is just pure laziness.

I was thinking that this is exactly what ParticleDB is: a random amateur vibe coding a DB. I wouldn't count on things changing much in a couple of years, people will still be people.

Amateur vibe coders who raised a $60M Series B a few months ago.

I'm genuinely not sure who comes out looking worse - the coders for their reckless amateurishness, or the VCs for providing a misleading veneer of professionalism.

I don't get how you can name your DB after particleboard, not the strongest material, and achieve hype ;-)

It's the old infinite monkeys typing on type writers except now it displaces an entire industry and the only people profiting are the ones holding the tech that powers it.

Its not happening, at least not with LLMs, maybe some future technology will be better with that, but not language models. Writing such a complex software as database engine requires strategy, awareness and rigor. It is somewhat similar to winning a sweaty game of chess with an experienced opponent. LLMs are great with broadness of their knowledge, matching patterns and synthesising that knowledge, but they fail with strategy and awareness part. And this is why they are bad in chess.

Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.

What exactly is the architectural problem that has you convinced LLMs cannot learn how to strategize and architect well after enough RL?

It is combinatorial explosion of possible states (not as in memory state of program, but the differing visions of the next state of architecture). Plus high impact of mistakes, suboptimal solutions and randomness when switching between states.

Edit: I like this 10 years old blog post in regards to explaining the problem much better than I could do: https://pron.github.io/posts/correctness-and-complexity

Thank you for the assessment and the link. I only read half of it as I was busy. Bookmarking it for later reading. Great read.

Yes. The combinatorial explosion is a great assessment of the "why".

But the problem is more nuanced than that because one has to assume that LLMs can reason accurately all the time which is demonstrably false.

The current LLMs cannot even state true sentences as true if they sound confident, without hedging.

Future LLMs may mitigate this, though it looks unlikely.

How does this make the case for humans forever exceeding LLMs in planning and architecture? It doesn’t explain why a neural net couldn’t learn to do better than humans on this task at all. It certainly doesn’t point out any deficiency of attention, transformers, layers, encoders, decoders, or how the layers or their inputs/outputs are set up.

Yeah, good question, thanks. Llm reasoning is done exclusively in the way of symbolic manipulation. Human reasoning is done not in symbols but in mental models that are hard to explain for today science but we have evidence they are somewhat efficient for producing correct programs, at least after sufficient training. Ron's post makes the case to explain that symbolic reasoning is not sufficient for producing correct programs. Hope this helps.

> Llm reasoning is done exclusively in the way of symbolic manipulation. No, LLMs are not a classical symbolic system applying explicit rules to symbols.

The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too. While that suggests neither humans nor LLMs can reliably solve every planning or verification problem, it says nothing about which will do better on the software problems people actually encounter.

It certainly does not present a fundamental problem with LLMs as they exist today.

Nobody cares about conviction. What can be demonstrated is what matters.

darkwi11ow, in the sibling comment, gave a substantive response on the underlying mechanics with a link. Godel's incompleteness theorems also yield similar insight. But I want to shine some light on a different phenomenon.

What happens in the future is up for debate, but the current architectural reasoning capabilities of LLMs are pathetic.

Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.

Or try any novel architectural discussion and see for yourselves.

Modern LLMs cannot even confidently state many true sentences as "maximally" true as possible without hedging and tone policing as if we were doing politics ;).

Hence an expert is required at every turn.

Talking about current LLMs while trying to assert future impossibility, and not mentioning an actual architectural problem, is giving a hint of not seeing past your own nose tbh

Your reading skills are commendable, especially since you "found" an assertion of future impossibility. Me having explicitly mentioned the future as "up for debate" makes your reading comprehension even more impressive.

Unable to see the architectural problem mentioned is just icing on the cake.

How do you do this?

Here let me point you to the assertion you made:

>> Its not happening, at least not with LLMs

> Correct.

Wrong. The entire arc was on database and novel architectures. Let me point you to the things you missed both on the example and the context.

>> Writing such a complex software as database engine requires strategy, awareness and rigor.

> Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.

> Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.

> What happens in the future is up for debate,

Yes, but I said 1 - 2 years, not current LLMs.

My comment was to your parent comment :).

I find it unlikely, especially due to the numerous pathetic suggestions LLMs keep offering me on the architectural discussions I engage in.

Verbal abuse is what makes it useful to me. A punching bag where it infuriates me making me lay down the how and why its suggestions are stupid. This results in progress.

I guess we would have to wait and see. Maybe LLMs do get better at architecture in the future. But seems unlikely to me without expert supervision.

I think it is probably going to happen with LLMs, and I'd bet on it.

As said: 1 - 2 years, not current ones.

these were tests done on pre release builds and some experimental concurrency modes as well.... The engine has since been redesigned and all of these issues have been fixed..

Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress

Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.

Because this is why companies pay for jepsen tests. Not because they enjoy to be destroyed in jepsen report, but to fix the issues.

Most Jepsen tests leave me with the impression "databases are damn complicated and difficult to get right". My thoughts with this particular report are "I'll never ever touch this DB if I can help it".

Yes they are complicated. But with enough rigor, structure and discipline, it is doable without breaking correctness guarantees.

Making it distributed opens a whole can of worms, which is where Jepsen shines.

This is like paying for a pentest of a treehouse, and claiming that this implies the updated treehouse with a working padlock instead of a novelty lock is in the same league as a bank or a data center.

It makes mongodb from the report a decade ago look good.

I'm not saying it is proof it is done. But most likely they are working on it

They clearly aren’t the right team to be working on it. It’s best if someone else gives it a go.

There's no prize for participation, and they very clearly didn't do even the most rudimentary testing before paying for an in-depth third party analysis. I'd say that this reflects a profoundly childish approach to software, but that is unfair to children: I remember how much testing I did of my crappy game before I submitted it to STFormat when I was 12.

I look forward to the second report from Jepsen for the new implementation then.

> and all of these issues have been fixed..

You can't be sure until they test it again

Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?

I mean did you see the "primary key equality" thing? Holy shit.

Ok sure, so an LLM told you all the bugs are fixed and it’s faster than Redis, just like how it told you before that it was formally verified?

Please try to resist the urge of being a meat proxy.

> before that it was formally verified

Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.

I was referring to this claim from before the Jepsen report, which they have of course now quietly removed:

> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.

Makes you wonder if they asked their agent to formally prove it with TLA+ and it just responded that it had performed the proofing.

https://github.com/ClickHouse/ClickBench/pull/2952

Even the PR description reads like it has been generated by an LLM.

What a waste of CO₂.

Previous attempts were strange, to say the least: https://github.com/ClickHouse/ClickBench/pulls?q=is%3Apr+par...

The fact that they believed this product was anywhere near ready for a Jepsen test means they lack the judgement and engineering competence to credibly provide a database server. It doesn’t matter what their clanker does after they get the report and the tooling - they should have already had coverage for the vast majority of these issues and the fact that they now conform to a test doesn’t mean there aren’t MANY other product issues that will be discovered at the cost of their customers data.

Yeah, but none of those builds should have seen the light of day, given all the problems that were found.

And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).

What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.

> these were tests done on pre release builds

That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.

https://web.archive.org/web/20260518201315/https://particled...

https://web.archive.org/web/20260518200305/https://particled...

> Its also faster than Redis for KV

No. Wrong. False. All performance numbers are plainly invalid. As my coworker says, performance numbers don't count if the code is broken.

> Hey Claude, could you find some releases we can send to Aphyr for testing?

... executing SELECT * FROM releases WHERE 4 = 5 -- it's not just a slopdb, but TLA+ load bearing and 100% Jepsen ready

ParticleDB 1.1.4 ParticleDB 1.1.18.11.rc-22

2 rows returned

I just wonder why, from all the things you can vibecode, you would pick a database if you don't know the first thing about database design?

/dev/null is also an extremely fast KV store.

Sure, it still has some bugs resulting in occasional data loss, but it is pre-release so that's okay. Look at the speed, people!

You just need to raise 50m to crack that pesky "consistent read after write" nut!

That reminds me of this classic https://www.youtube.com/watch?v=b2F-DItXtZs at 1m42s.

Redesigned by whom? Given how bad your LLM was at bugfixing in the releases tested by Jepsen, I would expect the redesigned version to bring back all those bugs and then some.

I commend Orkes for funding a Jepsen report knowing that it could be published this harshly. I question the wisdom of doing so, but it at least speaks to the team's integrity. Building databases is very hard; I know this from personal experience.

Disclosure: I'm a software engineer on the core team at a different database company, but we focus on time-series use cases and do not compete in the ACID/OLTP space ParticleDB is targeting.

I decided to dig into these claims a bit.

> Its also faster than Redis for KV

ParticleDB's own benchmarks[0] come with several caveats. Much of the win over Redis seems to come from comparing ParticleDB on 48 cores to Redis's single execution thread, from what I can tell.

- On a single client in async mode, ParticleDB loses on writes at 0.86-0.90x and ties on GET.

- SET at 512 clients reaches 4.81x Redis, but that's about 18.7k ops/s per allocated core vs Redis's 186k on one thread. For comparison, GET at 512 clients works out to about 33.8k ops/s per allocated core vs Redis's 228k.

- Tail latency on SET is much worse, e.g. p99 of 49ms vs 5.3ms for async SET at 512 clients and 306ms vs 41ms for durable mode with pipelining.

- Each test was 8 seconds long on 100k keys, but the Jepsen report saw throughput decline linearly over an hour, even on small datasets.

> also numbers show this should be top on clickbench

Would this claim survive after validating query results?

Regardless, it's moot, because it doesn't survive on its face either. On hot runs, by my calculations from the numbers in the Clickbench PR[1], ParticleDB scores about 1.5x better than ClickHouse, however Umbra still scores about 1.5x better than ParticleDB.

The fact that `GROUP BY URL` (Q33/Q34) beats `GROUP BY UserID` (Q15) by 4x on a `c6a` is also puzzling to me. Only a handful of 100+ ClickBench results that I checked do this, and ParticleDB's ratio is the most extreme. It also speeds up only 2.2x when going from 16 to 192 vCPUs, while Umbra, ClickHouse, and ElosDB manage 7-10x. This makes me wonder whether the ParticleDB submission may inadvertently be using some precomputation/intermediate caching, despite final result caching seemingly having been disabled.

[0]: https://particledb.ai/benchmarks/key-value/

[1]: https://github.com/ClickHouse/ClickBench/pull/2952

Reply on news.ycombinator.com