Cloth masks are somewhat effective at protecting those around you, but only slightly effective at protecting you; masks like the N95 do both quite effectively, if you wear them properly.
The biggest issue is that the minority of people like you are dictating how everyone else has to lead their lives. Dirty looks aren't "intolerance". Now, if they were telling you that you could not wear a mask that might be intolerant (see how that's similar to telling people they have to wear masks).