Is vllm on a gpu server. Unfortunately won’t be cheap if you need it 24/7
I'm wondering where this question keeps coming from and why people ask it - of course if I'm launching a public product it should be available 24/7. Is that what the question is?
>of course if I'm launching a public product it should be available 24/7.
Generally yes, but not a given. In some context like data processing you can accumulate tasks and run them in a batch.
e.g. OpenAI just launched batches...50% discount but you don't get instant results, you get them somewhere in next 24h
Typically when I think of a user using an AI product they're hitting the LLM a lot and expecting results on their screen right then, but I guess there are a lot of back office and data processing use cases.