It's so weird! Even weirder is the bit where you kind of have to beg the model to do what you want, and then cross your fingers that someone else won't trick it into doing something else instead.
llama.cpp ftw. It's not hard for it to be more productive than fighting with the absurd openai censorware... Sadly many of the instruct trained models are tainted with openai censorship because they used GPT4 output in the fine tuning-- but at least on those the trick of starting the correct output yourself works!
Also, llama.cpp now works (really well) with Radeon Instinct cards, which are stupid cheap because everybody thinks you need to buy nvidia stuff. Pcie bifurcation FTW!
() I know the basics of ML, NLP, transformers, etc. I know the theory, not even remotely close to how they really work.