It's similar to hearing someone say "um" or "like" a lot. It doesn't bother you much until you notice it, then you can't help but notice it all the time.

For me, "simply" and "just" are speed bumps in my reading. I notice them because they slow me down and serve no useful purpose in writing.