This paper combines knowledge distillation and efficient attention mechanisms.
=> It works (still efficient, lower cost).
Not an unexpected result, but to their credit, they established a new benchmark to test these combinations. KD+LongFormer is one of the best ones, retaining 95.9% of the performance for 50.7% of the cost.