We aren't reaching for it casually. Some of us consider privacy a fundamental value that must be defended, and regard an attack on our privacy with the same seriousness that we would regard an attack on our physical person.
Which is more of a danger to me, someone who punches me in the face on their drunk night out and gives me a bloody lip and a bit of pain for a few hours, or someone who betrays confidences that may have lifelong implications for my employability, insurance premiums and credit levels, ability to travel freely, and for that matter my self-respect and basic human dignity, before you even get to the kinds of more extreme and very physical dangers that could be posed by invasions of privacy if we consider the lessons of history?