If you have ever been in a math class with people who are not programmers, who are just plain old Joes, you'll notice that the most difficult subjects have to do with abstractions.

When particular concepts are represented by abstract concepts, and operations are executed on these concepts, that's where most people shut down. That's why many people don't go into programming - they are not able to easily deal with the abstractions that programmers have to deal with.

Languages that are similar to human languages - i.e, one word, one 'object' or 'concept' are easy for people to understand. A language like lisp is not even near to that - it depends completely on abstractions of concepts, and this is where the core difficulty for most people would lie.

For example, to take this example from wikipedia:

(list '1 '2 'foo)

The statement above is very difficult to reconcile with real world concepts, it's very difficult to put into the mental boxes we have grown up with.

Contrast with python:

if object is not null:

Languages don't have to be similar to english, but they have to be mentally mapable to models we already know. This is difficult to do with Lisp. With lisp, you have to understand new concepts. It's just not that easy for most people.