How do you decide what risks you can just leave open? Because that's what you're doing. The only thing that protects us from targeted rowhammer exploits for your hardware is our ability to research and write exploits ...
To go back with the human analogy. I have a server in a datacenter, there is a weakness in that someone can go to the machine physically and do nasty stuff, including hitting it with a literal hammer. This is a risk, but I can't do anything about it. Someone has to do the maintenance. But it can be mitigated with access control to the datacenter, among other things.
If the door to the server room can be opened with a toothpick, that's a vulnerability and it has to be fixed, because to "real" solution of not letting anyone in is not viable. I didn't chose to let people in the server room, it is just impossible not to.