Why would this be? Because AI allows you to copy your competitor in ways that was never possible before. It essentially becomes a skill and technology replicator. For example, look how Alpaca was able to somewhat replicate the multi million dollar models with $600 dollar cost.
https://www.home-assistant.io/blog/2022/12/20/year-of-voice/
I’d really like to start acquiring them. I know about The Pile and Common Crawl.
Are their any resources for individuals, that happen to have large amounts of storage, in regards to acquiring the best datasets being used?
I did lie a bit; OpenAI does have private datasets. For example, GPT-3 was trained on Internet data, but InstructGPT/ChatGPT was further finetuned through their own dataset of question/answer pairs. This was augmented in a really clever way: the initial set of valid completions was written by humans, but then OpenAI made more AI models to augment their training data before finetuning GPT-3 on it.
Even then, there's techniques for taking the output of one large language model and using it to finetune another. So just having a publicly-available AI chatbot is enough to get it to generate a dataset that you can then train another AI chatbot on.
I personally have been scraping Wikimedia Commons for public-domain imagery to train an art generator bot on; but this is mostly because I want an art generator that isn't full of pending copyright infringement lawsuits.
What I can’t get really pinned down is how much storage one needs to have your own copy of all of the most valuable raw data. At initial estimate it seemed like less than 1PB was enough for sure if focusing on text and not images/video. It even looked like maybe 200-400 TB might have been more than enough. I need to investigate more. I need numbers like 900GB for pile, then 2.5TB for a meta curated text only common crawl. So I’m assuming there is at least 10 of these in same ballpark that are important to have a copy of.
Any thoughts on the size requirements? Focusing on text.