Efficient LLM Scheduling by Learning to Rank

hao-ai-lab.github.io

2 pointszhisbug1 comment

We study how to approximate the famous shortest-job-first scheduling in LLM inference!