The rugby players that I know tell me that their pain limits how hard they tackle an opponent. They would not like to see padding introduced because a player would trust their armour overmuch and increase the risk of injury.
The upshot of this is that your head is involved in significantly less direct contact (though there's still some).
The OP mentions this, for example there are stern rules in the NFL against grabbing a player's facemask and cross blocks, but they still happen frequently. Furthermore, the games rules reward pushing the envelope (15 yards vs chance of serious injury to e.g., the quarterback is unequal).
You can make an effective argument that if you play football in the NFL you will sustain many serious injuries to your brain and body (as the OP does). I'd expand that to include college and high school football as well.