When I used "specific task", I meant specialized, domain specific tasks like financial sentiment and event extraction in which I hold a PhD. As a matter of fact, for Fiqa SA finetuned Roberta scores 88% F1 while BloombergGPT scores 75% F1. [1] Still very impressive for zero/few shot learner, but depending on data availability, performance targets and inference cost tradeoffs, it might not need your meds.
My point was "small" masked encoder transformer LMs like BERT can still hold their own on narrow domain tasks. And what OP claims that all NLP is solved by prompting a general purpose LLM service is simply inaccurate.