But to your point, I believe it was a YC partner (or three) who said that big businesses tend to start in a niche where a small number of people love you. If this guy really walked into a room and said "we've got a small, revenue-generating product with a positive viral coefficient and a growing user base", the line "fine, but you'll never beat Yelp" is not a compelling counterargument -- it's a lot like telling early AirBnB that they'll never beat Hilton.
I don't know if that's what actually happened -- there could be any number of reasons why a startup gets rejected after a ten-minute interview that aren't captured in a polite rejection email -- but in general, criticizing small startups for being small is not a winning strategy.