jhgg

When you consider the problem from first principles, understand how the CPU that runs the program actually processes the data, are aware of things like cache locality, vectorization, etc... and have good placement, preemption, bin packing, and defragmentation algorithms, you learn that in the limit, milliseconds is all you need.

When you are responsible for the scheduling of ~millions of GPUs, every second starts to add up, and you realize that doing this correctly very quickly pays for itself.

Consider a large training run that takes 200k GPUs, if it takes 5 minutes to schedule that job, create pods, etc, that's ~16.6k GPU hours wasted. At a market rate of $11/gb300 GPU hour, that's ~$185k of wasted compute in scheduling overhead.

This means that if say a job requesting GPUs is submitted with a given topology constraint, we would already computed the physical nodes that satisfy the topology requirement, and begun preemption / begun spinning up processes on the target machines within a few milliseconds of the job being submitted. If there were no preemptions required (which means we need to first wait for graceful termination), the job is fully scheduled in the blink of an eye, whether it's on tens or hundreds of thousands of GPUs.

At work, we have built something very similar, a custom topology aware scheduler for workloads, and the most interesting things about such a project are entirely absent from this article, making it more of a fluff/marketing piece than an actually interesting technical blog in this area.

Our next iteration of this scheduler aims to scale to O(millions) of GPUs, with a scheduling cycle time measured in the milliseconds.

This is rarely a problem in practice however.

    > government passes law that requires companies to age verify users
    > said government provides no way to actually verify a human's age
    > hilarity ensues

At my last job we resized a very large amount of images every day, and did so for significantly cheaper (a fraction of a cent for a thousand images).

Am I missing something here?

Our workaround was this: https://discord.com/blog/how-discord-supercharges-network-di...

The tooling would let us patch multiple modules at a time, which basically wrapped `:rpc.call/4` and `Code.eval_string/1` to propagate the update across the cluster, which is to say, the hot-patch was entirely deployed over erlang's built-in distribution.

League of Legends blogged about their updater here: https://technology.riotgames.com/news/supercharging-data-del...

I built something strikingly similar at my last job around the same time as well. Our server-side code was remarkably simple, just holding manifests (list of files and their chunk hashes) + a GCS bucket where all the chunks were stored (filename was just the hash of the chunk, making it content addressable.) Patching just involved the client understanding which chunks it already had, and which chunks needed to be downloaded from the server, or moved between files. Uploading a "new version" of the game to the server was also simple, just figure out what chunks were not in seen in the prior N versions of the game by looking at their manifests, and uploading them to the server, along with a new manifest describing the final contents of the game.

Given updating to a new version was just a matter of downloading new chunks, or re-arranging chunks that already existed on disk, going from one manifest to another did not depend on the new version being the successor of the prior version.

The majority of a game patch isn't actually the executable (where content-defined chunking wouldn't work well) instead were game assets like textures, audio files, etc..., which is why this strategy worked really well.

This means we don't have to deal with cache invalidation/consistency issues while also being able to handle thundering herds, for example a large server pinging @everyone and having a bunch of people click into the channel or launch their apps in response.