Hacker News Reader

Does anyone have any numbers on how long it would take to train an N-billion parameter ChatGPT model on K-trillion tokens, for some reasonably large values of N and K?

I'm guessing that at some stage, it would be hard(er) to keep these chips fed with training data. One would run into bandwidth issues keeping these monsters fed.

View on HN

You can take a look at chinchilla or Llama scaling laws. It depends on what you want to optimize. If you're interested in inference optimal, go with llama, if it's compute optimal, go with chinchilla. GPT-4 technical paper also has good scaling laws for how benchmarks will behave given extrapolating from a small training set.

Since the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.

Reply on news.ycombinator.com