Hacker News Reader

Comment by spongebobstoes | Hacker News Reader

thanks for your explanation. I'd love to see a concise explanation of the difference in benchmark performance between -preview and -beta

that would help clear the air, if the full-sql version has similar performance to the partial-sql version

View on HN

The new general version is actually faster than the version accused of cheating.

The architecture of the query engine is:

  SQL -> IR -> Assembly

-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.

If it delivers on being faster than the version accused of cheating with reasonable cache settings and no SQL feature limitations, I would say that resolves the accusations, regardless of the source code of -preview.

The remaining lesson here is that one should temper one's excitement with pre-release software, whether the software was built with AI or not (rather, especially if its built with AI). The preview results would have been more appropriate to announce on a social media platform than as a PR to the ClickBench repo.

The preliminary ClickBench results for -beta are in[0]. The geomean across all runs came out as 7% faster than -preview[1]

[0] https://github.com/ClickHouse/ClickBench/commit/56535d29e825...

[1] https://github.com/ClickHouse/ClickBench/commit/89cc76e1e4e2...

thanks, I think this is the most important point in your defense

Reply on news.ycombinator.com