That's completely false. There is nothing special about headphones, other than the fact that they make it easy to listen to loud music for protracted periods without annoying anyone else. Headphones are a serious concern to audiologists, but not because there's anything inherently damaging about headphone use, it's simply a very common way for reckless people to expose themselves to high SPLs. Good quality headphones can be used perfectly safely by anyone with the common sense to use them at an appropriate level.
If you're concerned that your use of headphones might be damaging your hearing, there is a simple and foolproof fix - purchase a pair of headphones fitted with a calibrated limiter. These are standard items in broadcast environments, where all-day use of headphones is commonplace.
In many environments, headphones actually function as hearing protection. A good pair of custom-fit IEMs or isolating headphones can attenuate background noise by 25-30dB. In many workplaces, you'll be reducing your overall noise exposure by listening to music at a reasonable level.