On the other hand, somewhere in the region of 10-15%[1] of the UK population is dyslexic (I'd expect that to be similar in any English-speaking country), so exclusively-written communication will be somewhere between "unpleasant" and "awful" for them.
That, and people will start judging on grammar, ability to spell, word choice, etc, etc. I think you'd just be trading one set of prejudices for another.