I think people who ride your train of thought just don't like religion and disagree with it being a fundamental right. To which I would vehementely disagree because most of the other important rights modern societies enjoy became a thing in a very religious time and religious reasoning was used (of course I have many more reasons as well). I think it boils down to what may seem like a false dichotomy but is very true -- you either take the route of regulating people's ability to live according to their belief or you don't.
Don't be surprised when this is used against you either. If you live in America or even some european countries (see poland and hungary) your irreligious practices become also regulated. If the state can regulate belief then it can for example force kids to pray at school or punish kids who pray at school. It took millenia of wars and conflict to finally decide, let's not tell other people how to believe and how to practice said belief. Let's not forget the past and even the present?