OS are designed to fully exploit available resources, Linux tries its best before triggering an OOM kill.
I recommend using the earlyoom if you want more aggresive oom kill:
https://github.com/rfjakob/earlyoom
The README contains a lot of interesting information.