Nowadays there's no lack of enthusiasts even in the smallest of niches. Just follow your passion and do what you enjoy, the others will be there. Much easier than trying to enjoy things you don't like, and building your social circles around that.
Of course even then you need to be active at least to the point of finding the others out there, but I'd still contest it to be the easier way.