I used to teach the first CS course at both Harvard and University of Washington. I didn't have this issue at Harvard, but at UW there was definitely a bimodal distribution in abilities. Roughly half the students had what I feel is a more visual / associative way of thinking vs. the focused structure you need to program.

The concept that the "non programmers" could never understand was the abstraction of functions. Like:

foo(a) { // stuff }

foo(b);

would confuse them because they'd get hung up on foo's parameter declared as 'a', but called with 'b'.