But I also wonder if there's something related to how people read that factors into this aversion to subtitles?
For me, subtitles are almost invisible and I spend zero-to-negligible effort "actively reading" them -- they just sort of get absorbed by my brain while I'm watching what's on screen. So it doesn't really negatively affect my enjoyment or engagement with the show/film at all.
I think maybe the folks that struggle with or dislike subtitles view the act of "reading" subtitles as a mental context switch that interferes with the parallel act of watching and listening.
Sometimes I do watch some TV shows with subtitles on (English dub & sub) if they have very inconsistent audio equalization i.e. during some scenes music is super loud and then they dip into really quiet dialog so I don't miss the beginning part.