I suppose that depends on what you're looking for. For example, for quite a few years my friends and I had an invite-only forum where we hung out, posted interesting links, etc. All told, about 30 of us at the peak. It was great, while it lasted.
A group of like-minded individuals could, and have many times, do something similar. Besides the problems you listed, you also have the one we eventually ran into: stuff gets stale after awhile. You have very few new members coming in, and various rifts appear in your community where people butted heads.
I'd also like to add that while as a community we do tend to be on the elitist side, we also tend to have a strong "info should be free" side, which may conflict with closed societies like this.
I think to really succeed here, we need to keep it public, but also keep the people who have already proven to be distasteful out. That was easier back in the day when people only had one email address if they were extremely lucky. Far harder today without requiring some sort of personally identifiable information, not to mention some way of validating it.