Generated Checklists Improve LLM Evaluation and Generation

The authors propose a novel approach where checklists are automatically generated to systematically assess and guide LLM outputs, ensuring more comprehensive and reliable evaluations by LLMs. E.g. it increases in the frequency of exact agreements between LLM judgements and human preferences from 46.4% to 52.2%.

From my perspective it would be neat if the benchmarks would support more model types and not only the predominat GPTs, which only showed that they can relatively easy be scaled up, though it was never stated that they can model language better with the same resources (AFAIK).