It would be interesting to see how a human researcher would do on the same task and same budget. 2 weeks of salary + a Claude Code max plan might outperform all of these auto-optimizers in a loop.
this would be an interesting experiment but i think one of the main benefits of having at least some automated pipeline is that human iteration is quite slow since the information intake speed is a few magnitudes slower. probably some combination would be optimal, like ensuring alignment of objectives at the beginning and tuning along the way