You could still argue that non-spectrum people are more "naturally" normal. Competition among societies has selected for groups where non-spectrum individuals are more numerous, possibly because this ratio leads to less conflict and more cooperation.
I wonder if groups are selected for a prevalence of 2 to 4 individuals per thousand [1] because these types of individuals have been responsible for technological progress in some way.