—————
Truly Open: Apache 2.0 license provides ungated access to weights and code. In addition, we are also open sourcing all of our data recipes and research insights.
https://github.com/Snowflake-Labs/snowflake-arctic/tree/main
https://huggingface.co/Snowflake/snowflake-arctic-base
https://huggingface.co/Snowflake/snowflake-arctic-instruct
To put it another way, when they share the weights for the model, that's like sharing the compiled output for some software - like releasing an executable instead of the source code that can produce the executable. They aren't sharing the things you need to produce the weights (the training code, training data, any preprocessing code, etc). Without those inputs you actually cannot even audit or verify how the model works. The team making the model might bias the model in all sorts of ways without your knowledge.
i) Open-Sourced Model Weights
ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/
iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly.
iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-serie..., https://medium.com/snowflake/snowflake-arctic-cookbook-serie...
v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.
It’s awesome the work you all have done. But not sure if I’ll return and remember the “we will” stuff, meaning that I’m not likely every look at it or start using it.
The source code is also apache 2.0
As an example of what open source actually means for LLMs, you can look at what AI2 does with their OLMo model (https://allenai.org/olmo), where each model that they release comes with:
> Full training data used for these models, including code that produces the training data, from AI2’s Dolma, and WIMBD for analyzing pretraining data.
> Full model weights, training code, training logs, training metrics in the form of Weights & Biases logs, and inference code.
> 500+ checkpoints per model, from every 1000 steps during the training process, available as revisions on HuggingFace.
> Evaluation code under the umbrella of AI2’s Catwalk and Paloma.
> Fine-tuning code and adapted models (with Open Instruct)
> All code, weights, and intermediate checkpoints are released under the Apache 2.0 License.
OLMo is what "truly open" is, while the rest is openwashing and marketing.
The GPL says it straight up:
> The “source code” for a work means the preferred form of the work for making modifications to it
Clearly just weights don't qualify, just like C run through an obfuscator would not count.
Most models that are described as "open source" are actually open weight, because their source is not open.
I get that there may be some training data that is proprietary and cannot be released. But in those scenarios, it would still be good to know what the data is, how it was curated or filtered (this greatly affects LLM performance), how it is weighted relative to other training data, and so forth. But a significant portion of data used to train models is not proprietary and in those cases they can simply link to that data elsewhere or release it themselves, which is what others have done.
I'd rather also have the data for sure! But in terms of what useful things I can do with it, weights are closer to source code than they are to a binary blob.
This is fully 100% OSI compliant source code with an approved license (Apache 2.0). You are not entitled to anything more than this.
The LLM equivalent here would be programmatically generating synthetic input or cleaning input for training. You don't need the tools used to generate or clean the data in order to train the model, and thus they can be propriety in the context of an open source model, so long as the source for the model is open (the training data).
No, the same way training is not a build dependency for the weights source code. You can literally compile and run them without any training data.
You mean the same source code? Because... I agree. That's why it's important for the source to be open. Both in the context of software and language models.