If you're going to add inflation, add inflation to _all_ your numbers. i.e: the cup of coffee will be subject to inflation, and what you save will also increase with inflation. If you run the numbers propagating inflation everywhere instead of cherry-picking, over 40 years you get:
total coffee cost: 84k
total investment: 220k
assumes 5% investment return, 2.5% inflation, and 20 coffees/month
spreadsheet here: https://docs.google.com/spreadsheets/d/1_-5SuX3mn_3GBQEOMoIs...
As for whether you should buy yourself a latte daily or not, do whatever makes you happy. Personally I think 220k in 40 years will make me happier, but I can also see a life-long morning ritual going to a coffee shop and reading the paper while being served a latte appealing.