Great result :)
Looks like the attention layers aren't as important as everyone thought because using a similarly sized feed-forward layer works almost as well (74% vs 77% top1 accuracy)
Looks like the attention layers aren't as important as everyone thought because using a similarly sized feed-forward layer works almost as well (74% vs 77% top1 accuracy)
If a network is the same size, less computationally demanding, and gives you 3% improvement, it seems extremely worthwhile, especially given the (mostly) diminishing returns of just adding more weights/layers.
Whether we need attention or not is a more interesting question on seq2seq models on text data.