It's weird that Lora and training with quantization is not being taken more seriously. It's way cheaper, takes less time, and a lot of evidence shows it's pretty good.
I don't think it should be something brushed on the side to be tried out later..
I don't think it should be something brushed on the side to be tried out later..
From a practical perspective, unless cost is really immaterial, I think most will end up starting with Lora, especially for 13b or 70b models.. you could do 10 fine-tuning runs for the cost of a few full fine-tunings.
But it's still all witchcraft to me to some degree, and I'd probably try full and Lora.