I think the "... is all you need" title here is particularly misleading as the paper does in fact use a BERT model for generating the vectors.
So if the implication was that no language model was needed at all and you can just do nearest neighbour on string similarity and patch results together, that implication was clearly wrong.
I think what the paper does show though is that there are methods that can make language models topic-specific without fine-tuning and that yield competitive results even with older models.