I think there are two big things:

1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for sequential processing (like in RNNs). Generally scales much better (see BERT/GPT-2/XL-Net/etc)

2. Separable convolutions. Significantly sped-up the standard CV networks. (mobile net, but now everything, really).

As a comparison, look at the transformer paper (also 2017), and the progress it has triggered.