Designing the metric of success is the hard part. The magic of LLMs is that they can learn a lot with unlabeled data and a relatively simple cost function. They can be trained the mountains of data we just have lying around without a ton of additional labeling or making clever domain-specific metrics.

If this approach doesn't really generalize well, it negates a lot of the usefulness of these methods and suggests we are further away than we might think.