While I don't agree with the French approach of regulating dress, one of their arguments is that you do affect other people with your choice of how to dress, and that people should not have to be subjected to certain overt expressions of religion in certain contexts, with schools being a particularly sensitive topic because of their position in passing on the ideals of the republic.

I think they go way too far, and a that some of it is bigotry hiding behind the excuse of secularism, but it illustrates that determining what sufficiently negatively "affects others" to be worth sanction can be used to reach widely different conclusions..