It is a drug much like... say, broccoli is a drug. Milk before bedtime is a drug. Running is a drug. It is a drug in the same sense that anything can cause addiction and withdrawal or be unhealthy.

Caffeine is, by itself, a drug in a different sense. It causes a slight addiction and can cause slight withdrawal. It isn't all that unhealthy in itself, however, especially not in the amounts we find naturally in foods. It becomes more of an issue if one is taking caffeine pills and other such things. Eating unhealthily might be more unhealthy, honestly - even though overall, having an unhealthy diet isn't a drug.