Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.
https://web.archive.org/web/20260518201315/https://particled...
https://web.archive.org/web/20260518200305/https://particled...
And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).
What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.
Making it distributed opens a whole can of worms, which is where Jepsen shines.
It makes mongodb from the report a decade ago look good.
Please try to resist the urge of being a meat proxy.
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
I mean did you see the "primary key equality" thing? Holy shit.
Sure, it still has some bugs resulting in occasional data loss, but it is pre-release so that's okay. Look at the speed, people!
You can't be sure until they test it again
No. Wrong. False. All performance numbers are plainly invalid. As my coworker says, performance numbers don't count if the code is broken.
Disclosure: I'm a software engineer on the core team at a different database company, but we focus on time-series use cases and do not compete in the ACID/OLTP space ParticleDB is targeting.
I decided to dig into these claims a bit.
> Its also faster than Redis for KV
ParticleDB's own benchmarks[0] come with several caveats. Much of the win over Redis seems to come from comparing ParticleDB on 48 cores to Redis's single execution thread, from what I can tell.
- On a single client in async mode, ParticleDB loses on writes at 0.86-0.90x and ties on GET.
- SET at 512 clients reaches 4.81x Redis, but that's about 18.7k ops/s per allocated core vs Redis's 186k on one thread. For comparison, GET at 512 clients works out to about 33.8k ops/s per allocated core vs Redis's 228k.
- Tail latency on SET is much worse, e.g. p99 of 49ms vs 5.3ms for async SET at 512 clients and 306ms vs 41ms for durable mode with pipelining.
- Each test was 8 seconds long on 100k keys, but the Jepsen report saw throughput decline linearly over an hour, even on small datasets.
> also numbers show this should be top on clickbench
Would this claim survive after validating query results?
Regardless, it's moot, because it doesn't survive on its face either. On hot runs, by my calculations from the numbers in the Clickbench PR[1], ParticleDB scores about 1.5x better than ClickHouse, however Umbra still scores about 1.5x better than ParticleDB.
The fact that `GROUP BY URL` (Q33/Q34) beats `GROUP BY UserID` (Q15) by 4x on a `c6a` is also puzzling to me. Only a handful of 100+ ClickBench results that I checked do this, and ParticleDB's ratio is the most extreme. It also speeds up only 2.2x when going from 16 to 192 vCPUs, while Umbra, ClickHouse, and ElosDB manage 7-10x. This makes me wonder whether the ParticleDB submission may inadvertently be using some precomputation/intermediate caching, despite final result caching seemingly having been disabled.
What a waste of CO₂.
... executing SELECT * FROM releases WHERE 4 = 5 -- it's not just a slopdb, but TLA+ load bearing and 100% Jepsen ready
ParticleDB 1.1.4 ParticleDB 1.1.18.11.rc-22
2 rows returned