People that learned and like LISP probably did so in the 1980s or 1990s when CPUs were much simpler, memory access times were constant, and "cache lines" weren't even a thing yet! Back then, randomly jumping around memory following a linked list was only about 2x slower than reading an array. Linked lists were much faster for some incremental operations, so the net result was that the two were very comparable.
These days arrays can be processed using AVX-512 instructions at a rate of 64 bytes per clock or even higher, whereas even one missed cache entry of a linked-list pointer will stall the CPU for several hundred cycles, during which time the array logic could have chunked through kilobytes.