Show HN: A simple scheduler for running jobs on GPUs | Hacker News Reader

Show HN: A simple scheduler for running jobs on GPUs

github.com

Examples simulated with GNU Parallel:

    simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt
    parallel -j3 --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt

    simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs}" -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2
    parallel --header : --shuf -j3 -v CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128

    simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs}" --n-samples 5 -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2
    parallel --header : --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1; seq() > 5 and skip() =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128

    touch gpu.queue
    tail -f -n 0 gpu.queue | simple_gpu_scheduler --gpus 0,1,2 &
    echo "my_command_with | and stuff > logfile" >> gpu.queue

    touch gpu.queue
    tail -f -n 0 gpu.queue | parallel -j3 CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' &
    # Needed to fill job slots once
    seq 3 | parallel echo true >> gpu.queue
    # Add jobs
    echo "my_command_with | and stuff > logfile" >> gpu.queue
    # Needed to flush output from completed jobs 
    seq 3 | parallel echo true >> gpu.queue

Interesting! Is there a way to use this, or some other library, that can spin up a GPU instance, on AWS or GCE, if there are no GPUs available, and discard GPUs when no more tasks are left? Maybe also allow some "overtime", in case a new task may get registered in the time when the instance gets free and a new one starts up?

    The package can simply be installed from pypi
    
    $ pip install simple_gpu_scheduler

When I run this, I get:

    ERROR: Could not find a version that satisfies the requirement simple_gpu_scheduler (from versions: none)
    ERROR: No matching distribution found for simple_gpu_scheduler

on Ubuntu 16.04. Admittedly I don't know much about Python or pypi so perhaps the issue is on my end. Nonetheless, I'm not 100% sure how to resolve this.

Try with hyphens instead of underscores? https://pypi.org/project/simple-gpu-scheduler/ has it both ways which is very confusing.

The README is incorrect, the PyPi package name uses hyphens.

Actually both should work. The problem in this case is probably that the package requires python3.6 and thus cannot be found using a python 2 pip. I changed the README for consistency though.

Sorry thats my bad. The package is written in python 3. Thus it is also possible that you have to install the package using `pip3 install simple-gpu-scheduler`. If the command pip3 does not exist try installing python3 using the ubuntu package manager apt.

    $ python3 -m pip3 install --user simple_gpu_scheduler
    /usr/bin/python3: No module named pip3
    $ python3 -m pip install --user simple_gpu_scheduler
    ERROR: Could not find a version that satisfies the requirement simple_gpu_scheduler (from versions: none)
    ERROR: No matching distribution found for simple_gpu_scheduler
    $ python3 --version
    Python 3.5.2
    $ python3 -m pip --version
    pip 19.3 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)

If it requires Python 3.6, then it's not compatible with Ubuntu 16.04 LTS out of the box which uses 3.5

Could you add an issue to the GitHub project? Shouldn't be a problem to add support for 3.5 and I'll take care of it soon.

Curious if anyone used a GPU-enabled Kubernetes cluster for something similar? I recently set one up. The only downside was an inability to assign fractional GPUs to tasks, so concurrent tasks were necessarily limited to the discrete number of GPUs available to k8s

This is one of the problems we solve at Algorithmia. We have a senior engineer set to do a lightning talk at Kubecon next month, on the topic: https://events19.linuxfoundation.org/events/kubecon-cloudnat.... If you’re interested in working on the problem, reach out to me.

We used to have a google doc where everyone would put name, machine, gpu(s), and estimated time to run, whenever they launched a simulation (longer than an hour). Low tech, but worked ok as long as people didn’t forget to do so.

Reminds me of how BOINC started out over a decade ago :)

A month in the lab saves a day in the library.

I also have the feeling that this should be "easy" with a traditional scheduler. But could you (or someone else) point me in the direction of something that's easy to set up for a quad-GPU machine? I spent two days trying to get SLURM to work, at one point. Currently I'm running with GNU Parallel, but it's much less flexible of course.

try PBS Pro.

My favourite scheduler is still "Son of Grid Engine", the open source continuation of the "Sun Grid Engine". I never had much trouble setting it up, but it is not really maintained at this point. PBS as suggested by others never worked well for me. But Slurm didn't give me much trouble, certainly not 2 days worth.

Son of grid engine definitely shows its age but I never had troubles submitting jobs to my school's cluster.

This is fundamentally why people buy DGX machines from Nvidia. It's preloaded. You have to figure out how to use them though.

Slurm works well even for a single machine. It does require some configuration but you only need to do it once and then it's very robust. The packages are available in standard repositories (like Ubuntu's) and there's a web-based configuration generator available. And if you ever want to add a second machine or more, it will be very easy to expand.

What stopped you from using Slurm?

Installing Slurm requires root access to the server/workstation, which is not always available. In academia the management of computational resources is often the responsibility of a separate IT unit which is not part of the research group itself. Further the IT unit might not be willing to implement desired changes to the server infrastructure or might take _extremely_ long to do so. Often the setup of a scheduler is also significantly more complicated in this context, for example due to authentication via a University LDAP or oddities of the server setup.

Luckily in our academic research group we administer our own workstations and servers ourselves. Has downsides as well of course, but at least we can fix or change anything we want hardware or software-wise.

I can imagine it would be excruciating to explain our requirements to an external IT department and then go back and forth clarifying what we want, make them learn stuff they'd otherwise not need, convince them that we really need it, wait until they have time...

The downside is that things are not "professionally set up" (we're computer scientists but not pro admins), but at least if there's a deadline and something isn't working, we'll surely fix it because we have skin in the game.

Slurm is not great. I found recently that when submitting an array of jobs, it will reject you if the upper bound is too high, regardless of the size of the array. I filed a bug but received no response.

That's according to documentation (https://slurm.schedmd.com/job_array.html).

> Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one).

Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit.

Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs.

And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.

It is documented, but it strikes me as simply being a lazy implementation. The consequence of this is that you cannot submit jobs 0-999, then 1000-1999, etc. You must submit 0-999, 0-999, ..., with an additional argument. It doesn't really prevent you from doing anything, it just gets in your way.

(The name MaxArraySize definitely makes it seem more like this is a bug than intended, albeit documented, behavior.)

I looked into submitting a patch but unfortunately MaxArraySize is used in more places than I would have guessed, and lost interest in trying to test all of them.

We're talking about installing Slurm for one's own use, so you can just change MaxArraySize in the configuration as an easy solution.

Did you actually _look_ at the repo? The author does not claim that this is the only way of doing it; and in fact, they seem to be quite aware of the existence of other schedulers. This is just a very simple one that does not require a lot of overhead and installation time.

In fact, I would go as far as to say that this scheduler is written in the spirit of Unix: make a task easy and provide a simple, chainable interface.

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

Reply on news.ycombinator.com