ssheng··on What is Azure thinking with this VM size NVadsA10 v5-seriesTo get a full A10, one must also get 36 vCPU and 440 GB of memory. I must be missing something.
ssheng··on Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLCQuality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
ssheng··on Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLCHow does Exllama rank among these? Heard good things about it.
ssheng··on Lamini Memory Tuning: 10x Fewer HallucinationsCreative idea. Any data on how much it takes to load the LoRAs and how much latency it adds to the generation speed?
ssheng··on Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TRT-LLM, and TGIHello! We're the authors of this blog post. Please let us know if there are other models and inference backends you'd like us to benchmark next.
ssheng··on Show HN: BentoML goes 1.0 – A faster way to ship your models to productionThe user feedback seems really encouraging. https://www.linkedin.com/posts/eric-riddoch_its-early-to-say...
ssheng··on Breaking up with Flask and FastAPI: Why they don’t scale for ML model servingFastAPI is great building block but can't expect it to work for model serving out of box.