His workers are poorer because his company's enormous valuation comes from the surplus value they produce but do not receive as compensation. You might be poorer if you own a small business those workers would frequent if they had more money.
Edit: You might also be poorer if you tried to compete with Amazon and were crushed like a bug by their anti-competitive practices.