It's reasonable, therefore, to suppose that there should be a "sweet spot" for caffeine use, where the tolerance you build is small, but the effects are still significant.
It's also worth mentioning that "tolerance" is not an all or nothing thing. With nicotine and amphetamines, there is a tolerance effect, but some of the effects of those drugs actually become stronger with regular use (and this is believed to play a role in the formation of addictions to those substances)[2]. So while tolerance and withdrawal are real problems, it is a myth that addicts merely return to the baseline in all respects when exposed to the drug they're addicted to.
In any case, if you are bothered by caffeine side effects, you might want to get some theanine. It's cheap, relatively flavorless, and when used in a 2:1 ratio with caffeine, it's been shown to reduce the side effects of caffeine[3], while adding some other cognitive benefits besides what you get from caffeine alone[3][4].
[1] http://www.sciencedirect.com/science/article/pii/00913057889...
[2] http://www.ncbi.nlm.nih.gov/pubmed/14592678