As long as we haven't reached AGI I feel like this is true of any new benchmark. BERT wasn't trained or designed for the task, give some smart folks a few months and they can now beat the task. The bigger question is what would we like an AI to be able to do. Is this benchmark a good one? Is there maybe a better choice of questions to get the type of NLP we want?