Because that's what it is. I spent a couple years reading /r/fantasy daily, and about once a month or so there would be a post something along the lines of "My SO/close friend/family member thinks I don't read because I listen to audiobooks. Are they right?" People believe there's debate about what they read there's debate about, and it can lead to accusatory comments like that. It's damaging when someone wants to be part of a community and feels like an outsider when they shouldn't.
> that can listen to a classic algebra math book with all the proofs and get the same learning experience as from reading and working through a paper book
1. I recently watched this video https://www.youtube.com/watch?v=WxDd8ocpcHk discussing what constitutes "valid" reading. Someone who reads 20 romance novels a month is a voracious reader just as much as someone who spends 3 years to get through Rudin. Reading romance novels is reading. Reading Star Wars novels is reading. Reading self-help books is reading.
2. Listening acuity improves with practice. If you listen at 3x speed (or faster, depending on the speed of the narrator), it's kind of incredible how many more words you can keep in your head at a time.
3. (This point is redundant because, see point 1) People attend math lectures all the time. Sure, there will be visual aids but who says you aren't supplementing with visual aids when you listen to an audiobook when you're learning something? And while I haven't researched it myself I'm sure there are a lot of accessibility options for listening to math that help a lot as well.