That's an unfair characterization. Their philosophy[1] says nothing of the sort. The gist of it is a focus on building software with "simplicity, clarity and frugality" in mind, targeted at "advanced and experienced computer users". There's nothing elitist about that.
It's a rejection of the trend of making complex, hard to use and maintain software. Have they gone too far, and end up alienating most users? Perhaps. But people like their software because of their goals and design sensibilities, not because they want to become part of some elite club. That's an old and boring argument people often apply to Unix/Linux users as well. Do better.