Some very interesting points and discussion here; what surprises me is no mention yet of the major thought experiment addressing many of the AI/intelligence arguments made here - the 29-year old Chinese Room paper by John Searle:

http://en.wikipedia.org/wiki/Chinese_room

In sum,

(A1) "Programs are formal (syntactic)." (A2) "Minds have mental contents (semantics)." (A3) "Syntax by itself is neither constitutive of nor sufficient for semantics." (C1) Programs are neither constitutive of nor sufficient for minds.