I agree but with limits: I have checked if I could use Jev for my code-review-pipeline. all the steps that could be done by Jev I already do deterministic and free for example which Files, which rules, which routes. I didn't need a classifier for this. The only real fitting slot would be the verify step itself who approves or disagrees to a finding.

But the actual point is that my generative part of this pipeline is slow and expensive. So to find problems and to explain them. And this will not be replaced by a classifier. I also think that the math in the article is dependent on how much of your pipeline actually has to make decisions.

What I also noticed that there is only US hosting and I specifically need EU hosting. It's also early access without SLA which I don't personally like. The thing with the calibration is that I first believe it when I tested it on my own cases. especially how often the verifier incorrectly rejects a correct finding.

This is the point where I agree with the article that there are only benchmarks from the company itself and not independent benchmarks.