Yeah, I see this argument all over HN. And in a sense it's true that the affront here isn't against math per se, but against general purpose computing.

But ultimately, the right to general purpose computing (ie, the right to run a program that doesn't snitch the plaintext) is the same as the right to engage in the underlying mathematics.

I think it's a distinction without a meaningful difference.