Model Size Compute Estimate
7b 1k GPU-Hours
13b 2k GPU-Hours
30/33b 4k-6k GPU-Hours
40b 8k-10k GPU-Hours
65b 10k-15k GPU-Hours
What would that mean in USD? Model Size Compute Estimate
7b 1k GPU-Hours
13b 2k GPU-Hours
30/33b 4k-6k GPU-Hours
40b 8k-10k GPU-Hours
65b 10k-15k GPU-Hours
What would that mean in USD?What you can do is fine-tune an open 7B model for a few thousand dollars, and that's the plan for these folks.
The article contains this:
We are currently seeking GPU compute sponsors for training OpenOrca on the following platforms:
* Falcon 7b, 40b
* LLaMA 7b, 13b, 33b, 65b
* MPT-7b, 30b
* Any other targets that get a sponsor. (RWKV, OpenLLaMA)
As I understand, a full round of training on the OpenOrca dataset would be comparable to going from LLaMa to Vicuna, but hopefully with more dramatic effects, if the techniques proposed in the "Textbooks is all you need" paper work as well as advertised.