(They did benchmark it against GPT-3 in general-purpose tasks and, unsurprisingly, GPT-3 came out on top.)
Press release link: https://www.bloomberg.com/company/press/bloomberggpt-50-bill...
(They did benchmark it against GPT-3 in general-purpose tasks and, unsurprisingly, GPT-3 came out on top.)
Press release link: https://www.bloomberg.com/company/press/bloomberggpt-50-bill...
We didn't compare to GPT-4, or any instruction tuned model. We're comparing a causal LM pre-trained only (BloombergGPT) to other causal LM pre-trained only models. You need to compare like to like.
Would an instruction tuned model work better? Sure! I would hope so. That seems like a good next step to explore.
It's also difficult to make scientific comparisons to models that we know nothing about. I don't know what data GPT-4 or GPT-3-5-turbo used for training, how many parameters they have, etc. We can't make any scientific conclusions when comparing two systems if we don't know anything about one of the systems.
We compared to GPT-3 results when they were available. Does GPT-3 do better on general purpose datasets? Yes, as we expected it would. It's a much larger model (~3.5 times the size). No surprise there. If anything, the surprise was how well bloombergGPT did against these larger models, beating them on some datasets.