I mean what’s to argue about really? If the product can do what I need it to do at lower cost and much lower exposure why wouldn’t I use it?
The vast majority of operations don’t need SOTA anyway.
The vast majority of operations don’t need SOTA anyway.
Also note that this is not a complete solution for self hosting. I assume that parts of this will be upstreamed to vLLM which this forked a while ago.
I agree. But people rarely pay only for the minimum they need.