> I think the only way to really do this well is to make HPC jobs a market system where resources cost money to the users. When money is involved, people are incentivized to optimize their workloads. But that’s rarely the case for large HPC clusters and I’d personally hate it if I had to deal with a HPC processing budget.

I also got that, there aren't many incentives in HPC. Users want results and don't have the time or skill to invest in rewriting the software stack to be as efficient as possible.

An intuition that I have is that this problem could be solved at the application/framework layer. Instead of launching jobs, which are glorified bash scripts, control the logic of the resource allocation dynamically from the application software.

It will solve the problem of load balancing based on your actual compute needs, but what happens if a program requests resources in a loop? At least the current schedulers know in advance the requested resources and can stall huge allocations