More generally, I think we've reached the point where the big players are investing a lot of resources and trade secrets into their giga-models. Having them effectively distilled and shared may expose these secrets or erode the competitive advantage. You can fine-tune an OpenAI model, but I don't think they are going to let you download the weights. I expect the same with Google and Microsoft.
It's not like these models are so big it's prohibitive for anyone else to reproduce the training, we're already seeing a profusion of open 1-11B param GPT-style models... the high-performing 770M T5 described here is smaller than that
> First, given an LLM and an unlabeled dataset, we prompt the LLM to generate output labels along with rationales to justify the labels
> Second, we leverage these rationales in addition to the task labels to train smaller downstream models.
This doesn't sound like online use of LLMs to me, but rather using them to pre-generate augmented examples.
At the end of the paper is this table:
Dataset Train Validation Test
e-SNLI 549,367 9,842 9,824
ANLI 16,946 1,000 1,000
CQA 8,766 975 1,221
SVAMP 720 80 200
OpenAI costs "gpt-3.5-turbo: $0.002 / 1K tokens"
let's say each rationale consumes 250 tokens ...that's $275 to augment 550M examples in e-SNLI dataset
Even if that estimate is quite far off in terms of token count, it seems like it'll only cost a few thousand dollars at most? GPT4 costs 10x as much.
It's more than I personally would ever want to pay as a curious bystander/hobbyist, but it doesn't seem like a serious "moat" to competitors.
Edit: just saw I wrote 550M in my original comment, even though I was thinking of it as the correct figure (550k)