That is not at all the behavior that I am observing on my system:

"time command ls -lR -color=always": 898 millis

"time command exa -lhgR --icons": 638 millis

This is in a directory of roughly 17k files of varying filetypes. Exa also outputs underlines and icons.

---

Admittedly if I create 80000 arbitrary extensionless files "ls" takes around 700 millis while exa takes around 1000 millis, but still the performance difference is nowhere near what you demonstrated.

I don't know what job you're doing that requires "ls"-ing >20k files, but on my system there seems to be no performance penalty before an uncertain, unachievable threshold.