But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.
I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.
Exactly. Once the price and performance get to the level where buying stuff for local training and inferencing… that is when we will start to see the LLM break out of its current “corporate lawyer safe” stage and really begin to shake things up.