Working countless hours on your startup project can also have devastating effects on your loved ones. They may be terribly upset. Your selfishness may be altering their future, and not for the better. (Using "you" in a generic sense here).
Only proves that working on a startup can be selfish too.