You are proving out his point. He focuses on only a few things that work for him and is not paying attention to other stuff. When you say no to most things then you will by your own decisions have a very narrow view/experience of the world.

That is the decision. Do you want to be super successful in one thing or a more balanced life.