Interesting and quite dense post. Singularity related articles and worries about "good" AI building makes me a bit uneasy when the performance of machines on many important tasks, e.g. object recognition, is dismal. So we have quite a way to go.

The usual rebuttal to the above is the "intelligence explosion" argument: once AIs can improve themselves, their progress will be exponential. As a tl;dr for the post, consider the Dewey summary of the argument:

  1. Hardware and software are improving, there are no signs that we will stop this, 
  2. and human biology and biases indicate that we are far below the upper limit on intelligence. 
  3. Economic arguments indicate that most AIs would act to become more intelligent. 
  4. Therefore, intelligence explosion is very likely. 

If you think about it, only (1) is a fact, the rest are assertions that are open to debate. For example, in (2) how do we define the "upper limits of intelligence", when even defining intelligence is problematic. Currently the human brain is the most complex and intelligent object we know. For (3), I don't know what sort of "economic arguments" are meant but as we all know becoming "more intelligent" is not a simple hill climbing process, in fact it's not clear how to go about it. Is Watson more intelligent than a person?

If you think about these facts for some time, you will find that AIs designing more intelligent AIs scenario has very little plausibility actually, let alone being highly probable.