This argument is a fallacy because it implies that money not spent is wasted somehow, but that is opposite of the truth, that savings underpin investment and thus growth.
"Growth" can represent too many things and is thus a great slippery beast that can be used to obscure helping the wider population. We continue to see economic "growth" but wage increases has been stagnant for decades. Owners need fewer workers to amass revenue due to changes in the type economy as well as automation. You can grow a business without benefiting as many people as you used to. Look at how many people Bell Labs had working for it when it was in it's heyday vs how many people work for Google now.
Google "asset price bubble" and compare this to the general inflation.