mdredze

Hi! I'm a co-author on the paper (Mark Dredze).

The arxiv paper is written for researchers who are building these models. We benefited tremendously from reading papers on GPT-3, PaLM, Chinchilla, Galactica, Gopher, Bloom, OPT, most of which are closed models. We are contributing back to that community, and the collective experience of people who are training these models. We learned a lot that will help others who are making their own decisions about how to train models.

If you are looking for an API to use, this isn't for you.

Hi! I'm a co-author on the paper (Mark Dredze).

"Doing math" is not really a goal of bloombergGPT. Take a look at our applications in the paper, which include information extraction, sentiment, reasoning, knowledge. These are models of language meant for use on text documents.

There are some aspects of the datasets that require numerical reasoning (ConvFinQA), but that's not the same as doing math.

Hi! I'm a co-author on the paper (Mark Dredze).

We didn't compare to GPT-4, or any instruction tuned model. We're comparing a causal LM pre-trained only (BloombergGPT) to other causal LM pre-trained only models. You need to compare like to like.

Would an instruction tuned model work better? Sure! I would hope so. That seems like a good next step to explore.

It's also difficult to make scientific comparisons to models that we know nothing about. I don't know what data GPT-4 or GPT-3-5-turbo used for training, how many parameters they have, etc. We can't make any scientific conclusions when comparing two systems if we don't know anything about one of the systems.

We compared to GPT-3 results when they were available. Does GPT-3 do better on general purpose datasets? Yes, as we expected it would. It's a much larger model (~3.5 times the size). No surprise there. If anything, the surprise was how well bloombergGPT did against these larger models, beating them on some datasets.