If you live in an urban area, I highly recommend walking to a coffee shop instead of using whatever you have in the office kitchen. While it costs money, I've found it a great way to take a break from work.
That's no reason to actually buy coffee though. You can just take a walk in the middle of the day.
However, caffeine is a larger problem. It's very additive with strong withdrawal symptoms, but giving it up for 6 months has significant mental health benefits. Unless you enjoy black coffee it often adds up to several hundred calories per day.
I agree the sugar and milk/creamer add a fair amount of calories to your diet.
However, because of how it integrates with your sleep patterns it can take 6 months for normal alertness levels to show up assuming total avoidance. Healthy individuals with reasonable sleep cycles should feel alert for the vast majority of the day without any form of stimulants.