This is not true. Think of the problems AI has with creating appetizing food imagery, for example. Images that every human annotator marks as unappealing with a high degree of consensus.

The problem is in the rewards, but it's more how they average across domains and contexts, so AI misapplies them. Actually, in human preference evaluations, most of Gen Z in particular extremely adverse to the common look of AI images.

So it's not the annotators involved in RLHF, it's the models failing to map to human preferences at the task level. The failure rate is also uneven across domains.

Where there are challenges with human annotation, it's not the ability to gauge what's good but rather how to correct what's wrong. Most people can tell when something is off but struggle to articulate why. That's where more domain expertise is required.