Hacker News Reader

Comment by kelipso | Hacker News Reader

To be fair, the open source model has been what's been working for the last few decades. The concern with LLMs was that open source (and academia) couldn't do what the big companies are doing because they couldn't get access to enough computing resources. The article is arguing (and I guess open source ML groups are showing) you don't need those computing resources to pave the way. It's still an open question whether OpenAI or the other big companies can find a most in AI via either some model, dataset, computing resources, whatever. But then you could ask that question about any field.

View on HN

But none of the "open source" AI models are open source in the classic sense. They are free but they aren't the source code; they are closer to a freely distributable compiled binary where the compiler and the original input hasn't been released. A true open source AI model would need to specify the training data and the code to go from the training data to the model. Certainly it would be very expensive for someone else to take this information, build the model again, and verify that the same result is obtained, and maybe we don't really need that. But if we don't have it, then I think we need some other term than "open source" to describe these things. You can get it, you can share it, but you don't know what's in it.

RWKV does: https://github.com/BlinkDL/RWKV-LM It uses „the Pile“: https://pile.eleuther.ai/ And I’ve seen some more in the last weeks.

Good to hear. Let's reserve "open source" for cases like that.

I agree with you to the extent that yeah technically it's not open source because the data is not known. But for these foundation models like Llama, the model structure is obviously known, pretty sure (didn't check) the hyperparameters used to train the model is known, the remaining unknown of data, it's pretty much the same for all foundation models, CommonCrawl etc. So replicating Llama once you know all that is a mechanical step and so isn't really closed source in a sense. Though probably some new term open something is more appropriate.

The real sauce is the data you fine tune these foundation models on, so RLHF, specific proprietary data for your subfield, etc. The model definition, basically Transformer architecture and a bunch of tricks to get it to scale are mostly all published material, hyper parameters to train the model are less accessible but also part of published literature; then the data and (probably) niche field you apply it to becomes the key. Gonna be fun times!

Keep an eye on the RedPajama project for a model where the training data and code should both be freely available: https://simonwillison.net/tags/redpajama/

These "open source" ai models are more like Obtainable models. You can obtain them. The source is not open, hence open-source. Somewhere open-source got lumped in with free or accessible. Obtainable makes sense to me.

That makes sense. But I would argue to smaller/cheaper models are not a threat to Google, they are a solution. They will still have the reach advantage and can more cheaply integrate small/low costs models at every touch point.

Reply on news.ycombinator.com