That's very well put. I noticed the issue myself when building my own evals and naturally started drifting into the direction you are describing, but you describe it so succinctly that now I am seeing how it should actually be done. Thank you
Yeah, I agree this approach can produce usable results. The more narrow and well scoped you can make each decision the model has to make (e.g. binary vs 0 to 100) the better the results you'll get (and the easier it is to troubleshoot and tune).