One is to commit memory on request. If all of your RAM and swap is committed, then malloc will fail. This is probably how we expect it to work, but it can be really wasteful. Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway.
Another way is to overcommit, where malloc doesn't actually reserve memory initially but only on first use, but only pause processes when memory fills up. If a process needs more memory and there isn't any more available, that process blocks until more memory becomes available. This means nobody gets killed, but it's not uncommon for the system to just deadlock like this forever once memory fills up.
And the third way is as you describe, to overcommit and kill processes when you run out of memory.
Note that there's no right answer, just different tradeoffs. Linux lets you configure this to an extent as well, so if you don't like the default, you can change it!