Forgoing a latte a day over 50 years means I essentially have to make 18250 decisions in order to save this money. But, on the other hand, let's say I decide to forgo, say, a $60k car and instead get a still very nice $45k car. If I invest that $15k in your hypothetical S&P index fund, and deposit no new additional money, after 50 years I'll have... $441,855.
So basically with just one, single decision I get the same benefit as someone that had to make 18250 decisions.
The small shit really doesn't matter very much. If you simply focus your frugality on a few simple things (namely housing, auto, and education) you will always be light years ahead of someone who is constantly worrying about their lattes or avocado toasts or whatever small daily bringers-of-joy.