Let me know in 30 years.
If you had more than 4% annual growth the whole time (on average), and the world grew only 4% (on average), yes you did make someone somewhat worse off. You may not know who that is, or where, or how many, but they exist.
edit: The one who you made worse off might be a billionaire, while you a middle class person. That's why I said it's a different question if it's a bad thing. But the fact remains.