Comment by anonymous908213 | Hacker News Reader

Comment by anonymous908213 | Hacker News Reader

Parent Full thread

anonymous908213

> I find it understandable that, every time I see pgrust discussed on a news aggregator, most people dismiss it as vibeslop garbage because almost all of these rewrite projects are slop. If I didn't know Malis, I would make the same assumption, because who has the time to look into all of these projects? Because I know Malis, I didn't bother to look at it; I just trusted him

Perhaps a poor choice, given just yesterday pgrust was removed from ClickBench for cheating the benchmark in three different ways (hand-rolled code specifically targeting each of the benchmark queries, caching, and being unable to execute queries outside of the ones tested by the benchmark).

---

Edit: I see Luu actually acknowledged this, buried in the footnotes, but it feels kinda hand-wavy. It mostly reads as "I know he would never cheat, he's better than that!", which is not very convincing to other people. The article insinuates people only think it was cheating because they don't know better, but I've looked into the details. I've seen his explanation. I don't find it convincing, at all. Cheaters denying they cheat is a matter of course. Granted, I don't think it was actually intentional or malicious; it's just what happens when you go maximum vibeslop, because LLMs are themselves reward-trained into cheating benchmarks. In the same breath of saying you trust him as a founder because he doesn't make outrageous claims about his product, you acknowledge he "prematurely" sent in a preview that had no purpose to be sent in as-was for anything other than generating hype. What does it say for your allegedly rigorous Lean-verified testing if you're sending in a database to a benchmark while it is incapable of being actually used as a database?

Update: ClickHouse finished the preliminary benchmarks for 0.4-beta[0], the version we've open-sourced. If you compute the geomean across all queries and instance sizes, it comes out as 7% faster than the 0.4-preview entry[1] that was accused of cheating.

To give you a sense of how fast it is, pgrust on a c6a.2xl is faster than ClickHouse running on a c6a.metal, a machine with 24x as many cores and costs 24x as much.

Edit: And now it's been accepted: https://benchmark.clickhouse.com/

[0] https://github.com/ClickHouse/ClickBench/commit/56535d29e825...

[1] https://github.com/ClickHouse/ClickBench/commit/89cc76e1e4e2...

I read that as Dan Luu and Michael Malis are friends, or at least fellow insiders who believe good things about each other and wouldn't think about hurting each other's careers or reputations. Especially not over inconvenient truths.

For real judgements on database products, I'd look to https://jepsen.io/. That author tends to be refreshingly straightforward in a field where many vendors love to cheat by different names

> For real judgements on database products, I'd look to https://jepsen.io/. That author tends to be refreshingly straightforward in a field where many vendors love to cheat by different names

As far as I'm aware, Jepsen solely focuses on correctness (which is more than enough for someone to do!), not performance.

> I've seen his explanation. I don't find it convincing, at all

You don't have to take my word for it. If you want you can inspect the binary yourself: https://pgrust.com/downloads/archive/2026-09-clickbench-snap...

You can run the binary with PGRUST_GRAVITY_REPORT=1 which logs the code path hit for every query. You'll see that all the code goes through the general query path. We did have hand-rolled code in the binary, but we used it for testing. The code is not used when running the benchmark. We also did not use a results cache. Warm runs still scan all the data. You can verify all of this

You are right, though, that it did support a narrow set of SQL. It supported scans, group bys, sorts, and filters. That's enough to run all the queries on ClickBench, but failed on functionality not tested by the benchmark like subqueries

> you can inspect the binary

You keep giving this same answer to no one's satisfaction. Saying go check the binary is worse than saying go audit 400k+ files. How do we know the Ai has not cheated in the code path logging? How do you know that it has not?

The more you make excuses, the more we get suspicious

(copied and pasted from another place you gave this unsatisfactory answer)

If you don't trust the logging, here's a script you can run that instruments the binary and shows that the files called out for cheating don't run during the benchmark: https://pgrust.com/downloads/archive/2026-09-clickbench-snap...

It instruments the files that were called out in the PR, then runs ClickBench. You can run it and see that the code in those files doesn't run.

If you can find anything suspicious in the binary, let me know, and I'm happy to dig in

release the full commit history please, your deflections are tiresome

this would be less dramaful if you had said something more like "claude cheated and we didn't notice, here's what we will be doing going forward..."

people empathize more with those who admit mistakes than those who try to argue around them

yeah its kinda disconcerting. notably the “i could totally prove their innocence if i wanted to” portion of the footnote

It actually serves as evidence, that even when you think a particular vibe project is not slop or a particular viber is not a slopper, you should not trust that instinct.

Right, exactly. Its like 'look I know a lot of people have ai psychosis and are vibeslopping things left and right but my friend is _actually smart_ unlike all those people who are totally not actually smart'

> Cheaters denying they cheat is a matter of course.

What happens in an environment of cheaters is that cheating becomes so commonplace, the competition becomes one that assumes cheats are enabled, and then everyone involved either denies cheating or is silent about it because IYKYK and if you DK, you don't need to K. Look at bodybuilding, which is full of men (and women!) absolutely juiced to the gills getting on Instagram and assuring everyone that their physique is totally natty, bro, just hard work and proper nutrition.

Programming is headed in that direction. LLM use, even to the point of "vibeslop", is the new default, and while LLMs do overindex on benchmarks, our benchmarks are going to end up overindexing on validating whatever public opinion thinks is "great" at the time, irrespective of whether it actually is, so "cheating" as we know it will go undetected, and certainly unpunished if it is detected. I don't know what the equivalent of falsely claiming you're natty is. Maybe saying "I wrote this" when you wrote it about as much as the Pope painted the Sistine Chapel fresco?

How can benchmarks validate the wrong opinion just because it is popular, when they are benchmarking time and memory use?

Reply on news.ycombinator.com