It's not like this hasn't been possible to do for many decades already.
I'm guessing the answer is something along the lines of "it makes applications freeze for <amount of time> and users don't like that".
Bu yes, the problem is probably not totally simple. adding delay in malloc while application might expect an immediate return could cause bugs. But swapping is a thing anyway and introduces delays so I don't know why they would not do this.
Also the tragedy of the commons, if everybody does this we are back to square one.
The actual solution is for the user to buy more memory, use a different app or change the workflow.
Also Windows should step in and say that this application is consuming almos all of the remaining memory, close others, or this or buy more RAM.
This is much more complicated that it appears, because the user could be intentionally doing that - if you have little memory (8 GB), you are sort of always almost out of it, if you have a lot but you are using professional apps like Photoshop, video editing, or doing machine learning with large datasets.
I'm sure that the reason Windows doesn't do this is because they couldn't figure out a good way to warn which is also not annoying.
There are some Windows warnings, I've seen them, but they only appear when the situation is so dire the computer is about to blue screen.
A marvelous operating system.
But yeah, at least I saw the warning fast, with the swap file it first came to a sudden halt, and after a lot of time it might have shown the alert.
I don’t see how you can realistically operate a computer this way unless you control all the software on it, or you enjoy buying way more RAM than you need.
(Also swap isn’t the only alternative to allocating memory - there’s also purgeable allocations and memory compression.)