Well-chosen symbols aid thinking about the concepts. That's why mathematicians don't write their formulas in ASCII. Once you learn the keyboard layout, it is faster to type APL than J's two-glyphs and three-glyphs.
Most people don't know about J, but know about APL because of those weird symbols.
I agree with your thoughts that not everything should be ASCII (and this is what many comments on this thread don't see), but don't reach the same conclusion as you do.
Well, not anymore. But I will prefer ASCII over glyphs any day.
I have learned symbols in Math, and I keep learning newer ones.
It finally comes down to _RoI_.
The vast and tremendous return I get from non-ASCII characters in Math (and have gotten for decades)- I will not get the same amount from APL. It's as simple as that to me.