- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.
Evals for robotics are also expensive.
- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data
- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).
It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).
The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.