Let's say I have a religion that, as part of practicing it, restricts my diet.

If I try to make you - a non-practitioner - eat that diet, then your reply makes sense. It's a personal matter; I don't get to force it on you or anyone else.

But if you try to remove the option of me being able to get food that fits my diet out in public? Don't try to justify that by saying "religion is a personal matter". That's an absurd rationale.