In regards to arguing that genetic defects cause lower intelligence, you have to define intelligence. The rate at which certain, particular categories of knowledge are learned? The speed at which knowledge is utilized? Are autistic savants the most intelligent people to have ever lived? People with down syndrome seem adept at reading and responding to body language, in my experience. Precise empathy is certainly a form of intelligence. I think our intelligences are diffuse. In the abscence of how biological data relates to these, we are just pattern matching.
After that, you have to ascertain what is even a factor in those intelligences: the total pool of genes we can draw from, the ways the genes express themselves on or off, or the delicate mosh-pit of biochemistry weaving our personality together. There are too many variables to control for, variables that aren't understood. It seems clear to me that environment dictates more than heritability of genetics. How much more is up for debate, but I'm not confident genes will be adequately contextualized in my lifetime.