As I see it, neurodiversity is more useful if the environment suddenly changes. In a stable environmental niche most forms of neurodiversity are probably a drawback.

For instance (reflecting on your examples) it should be obvious to us that the word 'crime' meant something much narrower several thousand (or hundred) years ago.