Hacker News Reader

george_123on Continuous batching to increase LLM inference throughput and reduce p50 latency

1) when the blog was released, TGI didn’t support paged attention, 2) many people don’t even know about TGI to reduce inference costs.

george_123on Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML

A lot of AWS services (especially SageMaker) aren’t very good in customer experience. People buy them for nominal capabilities and AWS core bread and butter — short-term and long-term reliability.

Most of these startups (AI and others) have to offer a compelling product before even being notable.

Besides, AWS top level doesn’t care if you use sagemaker or not. There’s a premium but if you’re still using EC2 via another startup, they’re still capturing lions share of value.

george_123on vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

george_123on Ant Group – scaling to 1.37M QPS on Ray

This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from previous year, and reaching 1.37 million TPS during peak times.

(resubmission, URL redirected to /Engineering last time)

george_123on [dead]

This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from previous year, and reaching 1.37 million TPS during peak times.