That's a piece of advice I keep hearing all the time, but I think it can be dangerous.
Of course, you have to start small - there usually isn't a choice not to. But, I think it's best to have a product which can serve a huge market in its most basic form without adding more features.
There are very few examples of startups which start with a product with features highly specific to one group of users and then expand out successfully to a huge market. Most startups die or get stuck in a vertical.