malisper

If they are the problem, shouldn't they be regulated?

> This is why Anthropic and OpenAI are pushing for more “regulation”

Given everything happening with Black October[0], don't you think regulating models may be reasonable?

[0] https://en.wikipedia.org/wiki/Black_October_(information_sec...

Update: ClickHouse finished the preliminary benchmarks for 0.4-beta[0], the version we've open-sourced. If you compute the geomean across all queries and instance sizes, it comes out as 7% faster than the 0.4-preview entry[1] that was accused of cheating.

To give you a sense of how fast it is, pgrust on a c6a.2xl is faster than ClickHouse running on a c6a.metal, a machine with 24x as many cores and costs 24x as much.

Edit: And now it's been accepted: https://benchmark.clickhouse.com/

[0] https://github.com/ClickHouse/ClickBench/commit/56535d29e825...

[1] https://github.com/ClickHouse/ClickBench/commit/89cc76e1e4e2...

The preliminary ClickBench results for -beta are in[0]. The geomean across all runs came out as 7% faster than -preview[1]

[0] https://github.com/ClickHouse/ClickBench/commit/56535d29e825...

[1] https://github.com/ClickHouse/ClickBench/commit/89cc76e1e4e2...

If you don't trust the logging, here's a script you can run that instruments the binary and shows that the files called out for cheating don't run during the benchmark: https://pgrust.com/downloads/archive/2026-09-clickbench-snap...

It instruments the files that were called out in the PR, then runs ClickBench. You can run it and see that the code in those files doesn't run.

If you can find anything suspicious in the binary, let me know, and I'm happy to dig in

> I've seen his explanation. I don't find it convincing, at all

You don't have to take my word for it. If you want you can inspect the binary yourself: https://pgrust.com/downloads/archive/2026-09-clickbench-snap...

You can run the binary with PGRUST_GRAVITY_REPORT=1 which logs the code path hit for every query. You'll see that all the code goes through the general query path. We did have hand-rolled code in the binary, but we used it for testing. The code is not used when running the benchmark. We also did not use a results cache. Warm runs still scan all the data. You can verify all of this

You are right, though, that it did support a narrow set of SQL. It supported scans, group bys, sorts, and filters. That's enough to run all the queries on ClickBench, but failed on functionality not tested by the benchmark like subqueries

Unfortunately, we have some sensitive stuff in our internal repo that we can't share. Fortunately, you can check the binary we submitted yourself: https://pgrust.com/downloads/archive/2026-09-clickbench-snap...

You can run it with PGRUST_GRAVITY_REPORT=1 and it will log the codepath that was hit for every query

On the lint, yes, I'm aware. It's one layer of defense. When we were prepping for the 0.4-beta release, we generated a large number of fresh queries and made sure the engine runs them and performs well across all of them

Fair point: the 0.4-beta source doesn't prove anything about the benchmarked binary. So don't take my word for it or rely on the source we published as evidence. Instead, check the submitted binary itself: https://pgrust.com/downloads/archive/2026-09-clickbench-snap...

If you run it with PGRUST_GRAVITY_REPORT=1 it will report the code path hit for each query. You can see that all the queries go through the general query engine

The binary does contain hand-written kernels in the irprobe files. These were used for testing. You can validate that when running the benchmark, those files are dead code

The engine supports a narrow subset of SQL: scans, group-bys, filters, and sorting. That's enough to run ClickBench. It doesn't support e.g. subqueries which is why some of the follow up queries failed

> This aged like milk. pgrust was caught cheating on all benchmarks this week.

Can you back this up? Our code is open source here[0]. You're free to audit it and benchmark it as much as you want. You'll find it's just as fast as we claim it is

[0] https://github.com/malisper/pgrust

My goal isn't to "prove" the software is correct. My goal is to find as many bugs as I can with the least effort. If I'm able to find all differences in behavior between pgrust and Postgres, sans bugs in the compiler, that's a real win for me

fwiw, I wouldn't say working at the assembly level solves the problem. You now need to make sure that your definition of the assembly behavior is correct

Happy to chat about it. I looked at Aeneas, but as described, it only supports safe Rust. I also need C support so I can prove both the C and Rust code

Right now, I wrote a compiler that takes C and Rust code and compiles it to Lean. Then I have an LLM write a proof that the C and Rust code behave identically. I've been able to prove equivalent behavior for about ~70k lines of C code with pgrust.

The issue I've started to encounter is that the context needed for each incremental proof has kept growing, and things have slowed down dramatically. My hunch is that the speed is the time it takes to prove a line of code is on the order of O(n^2) of how much you've proved so far

I'm looking to see if there are ways to shard things so the required context scales better. Today I was going through the formal verification work Anthropic and Theorem published a few days ago, and it looks like they used some sort of orchestrator to manage all their proofs and ensure good clustering of similar code. I'm seeing if I can get an approach like that to work

It's just difficult enough that I think it's doable if done right. Half of me is debating if I should wait until the next model release because any improvement in model performance will help a ton

This is roughly what I do in Rust. I use different static types for values based on how long they'll live. Temporary values get one type; cached values get another. The allocator allocates each value in a different location based on how long the value will live.

Yeah, that looks like a freelist approach. That will work as long as the values you're allocating are all fixed-size. You'll need something much more complicated if you want arbitrarily sized values

That example implements a linked list of integers. I'm curious how you would do a linked list of strings

Goose determines where the array will be copied to at compile time and allocates it there.

Rust does a similar thing. When a function returns a value, a location for it will be put in the callers stack frame. That means it automatically stack-allocates values when feasible. The downside in Rust is you need to know the size of the value at compile time whereas Goose can do it for arbitrarily sized values

From the link

> Keys and values are fixed-size

so not quite what I'm looking for

I'm curious, for someone more familiar with Goose: how do you represent an arbitrary number of dynamically sized values? If you have a fixed number of stacks, it seems like only a fixed number of values can change size at any given time.

For instance, how would you represent a hash table where the values are strings and at random, the program inserts or deletes a random string?

This doesn't seem like something you could easily represent. The only way I can think of is to keep a stack that holds all of your strings, and you periodically compact it.

Edit: Maybe you could also have a separate stack for strings of various sizes. One each for strings less than 8, 16, ... bytes. But at that point you're basically implementing your own malloc by hand.

Edit: Fwiw I do think Goose is very cool! I've had to hand-roll a lot of similar stuff in Rust before to speed up memory management

I'm not familiar with Goose, but based on some research and my understanding of how memory works:

At compile time, Goose will analyze your program and determine a number of stacks, N, for it. N is the maximum number of "growable" items you can have in existence at any given point in time. Using virtual memory tricks, each stack can grow up to 2GB, but it uses a contiguous address space and only uses as much physical memory as it actually needs.

For the program you're describing, I think you would end up with two stacks: one to parse the input and a second to print it. When you parse the input, Goose reads characters from the string one at a time and puts them onto the stack. Once it has hit the end of value, it will update the stack so it now represents the decoded JSON value (strings will just leave the characters in place, numbers will pop off the characters and push the number onto the stack). Goose repeats this until it has parsed the whole JSON, and now the whole JSON is on the stack.

Then, when you print the JSON, Goose uses the second stack to build a string. Like the parsing approach, it iterates through each element of the JSON and pushes its string representation onto the stack. Once it's done, it resets the stack pointers to free the memory.

This is really nice because allocating memory is just adding to the pointer, and freeing it is subtracting from the pointer.

In contrast, Rust automatically inserts malloc() and free() calls throughout your code, which are much more expensive than pointer arithmetic. On top of that, freeing a vector of heap-allocated objects is O(n), whereas if done right in Goose, it would be O(1)

No. I remember looking into this before, back when we were originally accused of cheating. I knew the code was dead and unused, but I got the reason wrong and had to double-check.

> This absolutely does not pass any smell test, by your own admission you included the test files "by accident"

If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark

I got it slightly wrong. It was code that you could run with an environment variable: PGRUST_GRAVITY_IRPROBE. It still wasn't intended to be part of the actual binary.

> The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else.

ClickBench tests a very narrow set of SQL: scans, aggregations, sorts, and filters. It ends up being a great starting point for building a query engine before layering in more functionality

Those are squash commits of many pieces of smaller work

The dev process carries over from when we constantly rewrote the codebase from scratch every few weeks[0]. We ran many experiments, most of which we threw away. A small number became part of "pgrust". It didn't seem worth it to maintain public branches with all of our development effort when we knew 80% of it was going to be thrown away

[0] https://malisper.me/postgres-in-rust-regression-suite/

I see what I can do. I'll need to audit it to see if we can release it

> This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview

That statement provided context and was not part of the defense. The point was that -preview didn't have the issues described, except for limited SQL support.

Yeah, it was a mistake to submit it as early as we did. I was very excited when we got the early version working and overeagerly submitted it to ClickBench

The new general version is actually faster than the version accused of cheating.

The architecture of the query engine is:

  SQL -> IR -> Assembly

-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.

I'll see what I can do. I'll need to go through and make sure there isn't anything sensitive in the source

It was included by mistake. It was code that was in our repo that wasn't explicitly marked as test code, so it got included in the binary.

Yeah, that was a mistake. We should have waited until it was more complete before submitting it.

We were just really excited about what we had built and how fast it was that we wanted to submit it.