I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks.
Trying to run bigger LLMs for an individual user to do big tasks is not going to be a good time.
I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks.
Trying to run bigger LLMs for an individual user to do big tasks is not going to be a good time.
A GPU is much like a hotel or an airplane. To make hotel and airplane economics work out, you need to keep your seats / rooms filled at all times. Because the number of spots is limited, an empty seat, room or request slot today is an opportunity forever wasted.
Unlike hotels and airplanes, though, GPU requests are short-lived. This means you need somebody to provide you with useful work to do 24/7. If you're an individual who needs to sleep or a single-country business that works 9 to 5, most of your request processing opportunities will go to waste. You're much better off renting your GPU from somebody who can provide it with useful work 24/7, and hence get much better per-request prices.
Idk what people were thinking. I’ve never seen anyone offer a plausible way to sidestep batch processing for example.