What is the "standard linux file system"?

ext4 on an old system, feeble in comparison to yours, performs much better.

ext4, 8GB memory, 2 core Intel i7-4600U 2.1GHz, Toshiba THNSNJ25 SSD:

$ time ls -U | wc -l 555557

real 0m0.275s user 0m0.022s sys 0m0.258s

stat(2) slows it down, but sill this is not as poor as your results:

$ time ls -lU | wc -l 555557

real 0m2.514s user 0m1.126s sys 0m1.407s

Sorting is not prohibitively expensive:

$ time ls | wc -l 555556

real 0m1.438s user 0m1.249s sys 0m0.193s

Drop caches, sort, and stat:

# echo 3 > /proc/sys/vm/drop_caches

$ time ls -lU | wc -l 555557

real 0m6.431s user 0m1.249s sys 0m4.324s