Sure, I wouldn't expect most computer users to learn TeX/CVS/etc. I wouldn't even expect a majority of new hackers to learn it any more. But for those that do learn it to a sufficient level of mastery, it's not only usable but at times wonderful.
I think there are actually two "problems" going on at the same time. On one hand, open source software is "too difficult" for most people to use. I agree. But on the other hand, most people aren't putting that much effort into learning the software.
Is software supposed to be easy to use? As much as possible, I would think. But some "hacker" software that has a tremendous initial learning curve (TeX, Emacs, even Lisp, depending on your background) offers equally tremendous rewards over similar software that is easier to learn.
The reason, perhaps, is that in learning TeX, you are learning how to prepare documents at a more abstract level than Microsoft Word. The resulting documents may be largely identical, but TeX offers the user more power and control in exchange for taking the time to learn how to operate it. This is not necessarily a bad thing, especially if your job is to prepare documents. It may be a totally awful thing if you only need to prepare an occasional document, as the benefits of TeX are not likely to be worth the effort required to learn it for you.