I recently quit caffeine. Like you, I never noticed a difference...until I took my workouts more seriously. Recently I noticed that caffeine makes it more difficult to go hard at the gym. My heart rate spikes more readily.
I use a pre-workout from a very famous supplement brand which includes 175 mg of caffeine and I can definitely tell the difference. I can't tell how much of that is because of the caffeine, because it has more stuff like creatine and L-carnitine among other things, but I cant I feel like I last more doing this and I have more energy.