Yes, I don't understand why they are using a search benchmark for these... it would be much better to have something like giving it a story up to the context length (from a book? how to find one that's not in the training data?) and have it write a new chapter/ending that is consistent with all prior text and introduces zero inconsistencies.
But how can you automatically evaluate whether it did this?