Few (non-immigrant) families eat it, therefore few people know how to cook it properly. And so when they decide to try lamb on a whim, it comes out tasting bland, they say, "Ugh! Lamb is awful!", and never try it again.
It took me over a year to help my girlfriend un-learn that initial reaction by repeatedly taking her to Mediterranean and Indian restaurants where they actually know how to cook lamb.