Thanks for sharing, it was really good read. The authors in the article have done a good job for sure to validate their hypothesis and make it work for a particular subset of problem.

1. Local hardware, no networking or HTTP requests 2. No other parallel requests 3. 1000 runs bounded by 1000 tokens

I have worked a little bit in academia and I am not a big fan of the way favorable samples for the hypothesis are kept and unfavorable ones are thrown away. I might be wrong here, however its highly unlikely that the team would have just worked with one query. Chances are that a lot of different prompts with varying number of runs would have been tried to see what works and supports the hypothesis.