You are indeed making the “things that work in other places won’t work in America because of differences in race and ethnicity” argument.
This is a fairly lazy analysis that plays on America’s often unspoken cultural beliefs around race. While it is true that race is a major cultural marker in America, this fact is not innate and the oft repeated argument that “homogenous” societies can do things that we can’t amounts to little more than “we can’t do it because we are different”.
The determinants of diversity in the sense that it might be relevant to a discussion on free speech depend on a multitude of cultural factors. You can’t just say “well they aren’t diverse because they don’t match the criteria set out and defined by _America_”.
95% of Americans speak English. If half the country didn’t speak English well that would make for a very different society and culture than the one we have now.