(Cf. BloombergGPT paper all financial benchmark tasks).
And that's not even taking into account inference cost, but that is a business case issue.
(Cf. BloombergGPT paper all financial benchmark tasks).
And that's not even taking into account inference cost, but that is a business case issue.
I'm not sure what BloombergGPT has to do with LLMs vs non-LLMs; BloombergGPT is an LLM [2], and it defeating other LLMs on financial benchmarks doesn't prove much about large language models other than "LLMs can be trained to be better at specific tasks."
1: https://arxiv.org/abs/2005.14165
2: https://www.bloomberg.com/company/press/bloomberggpt-50-bill...
When I used "specific task", I meant specialized, domain specific tasks like financial sentiment and event extraction in which I hold a PhD. As a matter of fact, for Fiqa SA finetuned Roberta scores 88% F1 while BloombergGPT scores 75% F1. [1] Still very impressive for zero/few shot learner, but depending on data availability, performance targets and inference cost tradeoffs, it might not need your meds.
My point was "small" masked encoder transformer LMs like BERT can still hold their own on narrow domain tasks. And what OP claims that all NLP is solved by prompting a general purpose LLM service is simply inaccurate.
I hadn't read the financial paper you linked, it's very interesting! One odd bit I did notice was they set the gpt-4 temperature to 1.0, which is... not a great setting for analysis, and probably harmed the results somewhat. Typically you'd want a value much closer to 0 for that. But while a lower, more reasonable temperature setting would probably improve gpt-4's performance, I would still expect a finetuned LM to outperform larger models with just prompting on those kinds of narrow domains, especially once cost is a factor.
It's somewhat surprising to see how bad Bloomberg-GPT was... Even gpt-4 trounced it on every published metric, and it wasn't trained for finance tasks specifically. The bitter scaling lesson, I suppose.