A well-fitting isolating IEM can slice off 25dB of noise, more than enough to (mostly) eliminate background conversation noise at any location short of a hip restaurant. These IEMs tend to be so sensitive that they become earsplitting at a tiny fraction of the maximum output volume, way less than 60%.
Learn approximate dB levels of various sounds, compare various noise with a meter (even an uncalibrated app will give you a ballpark idea of what you're dealing with), and listen to music at an average 65dB or less (this will obviously vary with dynamic range of your music; occasional 80-85dB peaks won't kill your hearing).
One trip to a dance club or rock concert without earplugs (100-110dB on average in my experience, 120+ has been known to happen) will do more hearing damage in a few minutes than a lot of headphone listening while working. I'm pretty sure the busy street near my home routinely hits a 90dB average at rush hour, solidly in the danger zone compared to reasonable headphone use.
Edit: To learn more, read the following: