HNHacker News
TopNewBestAskShowJobs

ActivatedAI

13 karma · joined August 2, 2024

LLM focused research lab in SF.

Cofounded by a ML eng at Google with 16 years of experience, and an LLM obsessed brilliant hacker who was using the GPT3 API on the first day of its release.

submissionscomments
ActivatedAI··on IsoFLOP curves of large language models are flat
Page 9 on the Llama tech report has an interesting graph that predicts task level performance from the cross-entropy loss. The sigmoidal model fits well, and at the steepest part of the S, a .01 change in NLL is worth about 5% task level accuracy.

Here is a quick screenshot if you are lazy.

https://snipboard.io/C6mipQ.jpg

And here is the paper if you want to dig deep into the highest quality published research on LLM frontier.

https://ai.meta.com/research/publications/the-llama-3-herd-o...

ActivatedAI··on IsoFLOP curves of large language models are flat
There is some good published research about doing multiple passes over the training data, and how quickly learning saturates. The TL:DR is that diminishing returns kicks in after about 4 epochs.

https://arxiv.org/abs/2305.16264