The point is caffeine etc. corrupt the mind and cause a person's mental faculties to run in a way they were not initially designed to.
The point is not that these drugs are all extremely harmful, only that they are all harmful. Caffeine and other things get a pass because the "hard drugs" are so uniquely and visibly harmful that they overshadow all other forms of harm.
One could even say that this has tricked us into thinking that lesser drugs like caffeine or canabinoids are "effectively harmless" because they're not causing us to OD or steal things to get another hit or causing visibly psychotic states. But that is not true. We've simply accepted that the harm they due is not worth thinking about (this is subjective, not objective).