If you don't like the fact that a VC has the potential to make more money than you, the solution is to not take investment from one, not to ignore them after they take investment.
Also remember that yes, they're not putting in a lot of work to make the money - but you aren't putting in a lot of money that you might lose, which they did.
I'm not arguing for or against working crazy hours, just against the idea that you should be scared of any suggestion from a VC.