From all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases.
What use cases do people have for these smaller LLM's?
What use cases do people have for these smaller LLM's?
The metrics are good though, perhaps placing this closer to 13B.
And 8K context is huge. When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM consumption for a ~13B model.
QLORA is the gold standard for more affordable training.
As for datasets, just look at the open datasets the best-in-class models are using, like Vicuna or https://huggingface.co/NousResearch/Nous-Hermes-13b
Some model datasets like Manticore, Chronos or the infamous Pygmalion are more "secretive," but you can find the dataset gathering scripts on Github or in community chats.
You can easily finetune 7B or 15B LORA model with that on consumer GPUs.
It's a rabbithole, and unfortunately there's no good shortcuts.
There are startups that do finetuning on your own data, but with zero hints on how to preprocess your data and absurd costs (both upfront training and GPUs for serving inference) that's it's extremely difficult to argue from a customer business perspective compared to just using an API.
Vapourware GPT startup inc is valued at $2bn the afternoon after you form the company and buy your first macbook.
Actual usage of Ai, fine tuning etc. I can offer you $100,000 for 30% of your company if you can demonstrate a fully working product.
If all you have is an M1 or whatever, ya, you need a real workstation and depending on your use ChatGPT might be cheaper/better.
You still need the memory to be able to go that high, but it's totally doable.
But I assumed full training would give better perplexity for large contexts, and perhaps this method would be more effective at 16K+ with an 8K model to start with.
None. Training a functionally useless model and releasing it is a great way to demonstrate that your company is hip and current. That way when prospective clients ask about AI you can vaguely gesture at some model that you released and say you employ cutting edge AI experts.
If it was good, then they’d charge for it.
…what they (and everyone) is gonna do is play with smallish models to iterate on the process for relatively small expense and earn karma.
Then pay big $$$ to make a really good model for internal use and/or an api that people have to pay for.
Tldr; it’s free. It’s by sales force. You should expect it to be a) crippled and b) a loss leader for a paid product.
Not judging; it’s a fair strategy. Just saying: salesforce is not a company that just gives hundreds of thousands of dollars away for nothing.
If you want a good free open model, you’re kidding yourself if you think a corporate giant is going to kiss you on the head and give it to your for free.
Yep! That makes sense!
I would love to be the CEO of the company that does give away an actually useful model and little forehead kisses though. The amount of goodwill that one would generate from that would be astronomical and training costs are getting so low that nearly any company with enough cash could do it.
I look forward to waking up and hearing that the Nabisco/Canadian Tire/A&W usefully-tuned model is revolutionizing the economy and seeing the infinite amount of good press that it would generate.
The problems with chatGPT are many - dependence on third party, privacy, externally imposed ideology and rules, cost, and most importantly - prompting is context-size limited and token-expensive, you can't pack much data into it.
Fine-tuning is a more powerful approach where you can actually fix the model problems instead of futzing around with the prompt and demonstrations. Yes, you got to work on your dataset. But if you don't already have it you can bootstrap with GPT-4 for a small sum.
Meta provided the training wheels - LLaMA, every company tried fine-tuning it for their purposes, but could not proceed for lack of a commercial base model. Salesforce XGen and a few other open-small-LLMs (funny how that sounds!) open the flood gates.
So the recipe is: use an existing dataset, or make one with regular GPT-4 prompting and a bit of curation. Then fine-tune a small open model. You can get it to be better than stock GPT-4, cheaper, faster and private. If you use LoRA's you can save each skill in a separate diff model just 1% the size of the base model and use a single GPU to fine-tune it, in a single day.
These free models are both a defensive move against behemoths, and kindling to rapid business development.
You don't have to go to openai.
https://www.reddit.com/r/LocalLLaMA/wiki/models/ gives you a list of VRAM requirements to load the model into GPU VRAM. the more VRAM the computer has, the larger the model you can load in, thus making 3090s the current consumer grade king due to price to max VRAM.
This being said however most models are LLAMA based which all fall under that specific research license.
So following the rules, you would be limited to a subset of models which are foundational models which allow for commercial use
I can barely run 33B, but anything more than 800 context and I oom. But it would run very comfortably on a bigger GPU or a 24GB+ laptop.
Theoretically some phones can comfortably handle 13B on mlc-llm though in practice its not really implemented yet.
If you're relying on prompting for the 7B models IMO you're gonna have a bad time — they're mostly toys at that: interesting output but not consistently useful. But finetuning gets better results, and it's cheap to finetune.