Hacker News Reader

Comment by jhgg | Hacker News Reader

This blog post spends a lot of time telling why gang scheduling is needed here (a fact obvious to anyone using the nvl72s), but doesn't actually go into detail about how their scheduler works.

At work, we have built something very similar, a custom topology aware scheduler for workloads, and the most interesting things about such a project are entirely absent from this article, making it more of a fluff/marketing piece than an actually interesting technical blog in this area.

Our next iteration of this scheduler aims to scale to O(millions) of GPUs, with a scheduling cycle time measured in the milliseconds.

View on HN

Feels AI-generated, so much text real estate is spent explaining the basics in sufficient detail then just two paragraphs on their scheduler with a lot of hand-waving.

This blog's goal is to introduce the problem around the new NVL72 system with GPU failures in production, which not all people outside of xAI are aware of. We have more blogs coming up for what we built to handle topology-aware scheduling, defragmentation and hot swapping during failures. The short version of the scheduling piece: jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps.

Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"

> making it more of a fluff/marketing piece than an actually interesting technical blog in this area.

/genuine do you recommend any other blog?

Why do you target millisecond-level cycle time? Are you looking at very fine-grained tasks like elastic sandboxes, in addition to beefy pre- and post-training jobs?

Because inherently if you build this software correctly, you get millisecond level cycle times.

When you consider the problem from first principles, understand how the CPU that runs the program actually processes the data, are aware of things like cache locality, vectorization, etc... and have good placement, preemption, bin packing, and defragmentation algorithms, you learn that in the limit, milliseconds is all you need.

When you are responsible for the scheduling of ~millions of GPUs, every second starts to add up, and you realize that doing this correctly very quickly pays for itself.

Consider a large training run that takes 200k GPUs, if it takes 5 minutes to schedule that job, create pods, etc, that's ~16.6k GPU hours wasted. At a market rate of $11/gb300 GPU hour, that's ~$185k of wasted compute in scheduling overhead.

> O(millions)

Big-O doesn't include constant factors for a reason (never mind just being a constant entirely), but maybe I am misunderstanding what you are trying to say?

"Order of millions of GPUs being scheduled on" - which is to say when a workload is submitted to the scheduler, it makes a scheduling decision and the individual compute nodes are being programmed within milliseconds.

This means that if say a job requesting GPUs is submitted with a given topology constraint, we would already computed the physical nodes that satisfy the topology requirement, and begun preemption / begun spinning up processes on the target machines within a few milliseconds of the job being submitted. If there were no preemptions required (which means we need to first wait for graceful termination), the job is fully scheduled in the blink of an eye, whether it's on tens or hundreds of thousands of GPUs.

You are abusing big O which is worst case time complexity scaling. If I simplify your notation, then you just wrote O(1)

Should have said "order of"

It's not Big-O complexity notation. It means "a few millions". It's sort of how the word "sanction" can mean one thing or a different thing. The textual glyph sequence means a different thing based on context in English. Pretty normal.

I like to call it Big-O Magnitude Notation.

why not just say 'millions'

"Our next iteration of this scheduler aims to scale to millions of GPUs"

I treat it as more akin to the (slightly more-awkward) "single-digit millions" whereas "millions" could go over. It's just a SV-ism. I usually find it pretty easy to disambiguate and I confuse it as little as I do the same / in dy/dx vs. 1/2. Notational overload is pretty normal in human communication, I think.

It means ~millions

open source the scheduler

jhgg and jlf PLEASE OPEN SOURCE THE SCHEDULER

i suspect you also have some great 60Hz cluster observability software ;)

its good for the world if you do it

Reply on news.ycombinator.com