I think this kind of defines the line between an addict and a user, doesn't it? If you're an addict, you can't just 'take it or leave it', or 'just stop'.
After going through opiate withdrawals, I found quitting smoking rather pleasant in comparison. It's really not a ride you wish to take again.
I've also done lots of amphetamines and cocaine, and never felt like I was getting addicted. Maybe there's a genetic component.