But it has some kind of direct replacement mode, why aren't there benchmarks for that?
Crickets...
That's not fair. They've published a paper which has lots of details: https://arxiv.org/pdf/1711.04422.pdf
> Performance of the SPEC benchmarks was also unevenly affected by Souper: five become faster and seven become slower. None of the differences was very large.