Thanks Lucas, appreciate the feedback as always!
76 karma · joined November 6, 2023
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.