(afaik) it does not support batching, and some other server focused features.
I was talking more about a high throughput server, which is not appropriate for ollama or llama.cpp in general.
I was talking more about a high throughput server, which is not appropriate for ollama or llama.cpp in general.
No comments yet.