A place that allows discrimination is not a positive environment for everyone who is not part of the discriminated group.
People who are not part of the discriminated group do see what's going on. However they are often powerless themselves to do anything other than leave. A toxic environment affects almost everyone exposed to it.