This just seems like a confusion to me. My job is the time I spend getting money to pay for the things I need to live. By definition, the work I do then is of value to someone paying me, but not to me. The rest of my time I spend doing things which I can choose, including things I think are a positive contribution to the world.
I could make sense of arguments such as:
1. You should consume vastly less, so that you spend less time earning money and more time contributing to improving the world.
2. Society should be organised differently so that people have to work less and can contribute as they see best in their increased non-paid-work time.
3. You should spend a lower proportion of your free time doing things that benefit yourself and more doing things which benefit the world.
But "you should spend the time you dedicate to getting food, clothes, energy and shelter to contributing to good causes?" Doesn't really make sense.