I suspect it also varies between drugs in the same person. I've been using temazepam (as prescribed) for years, and if I don't take it for a while, it's not a huge issue - I don't crave it, and I don't have any withdrawal symptoms. Similarly with opioids - I was using them for months due to chronic pain (codeine or tramadol, again as prescribed), and when I was advised to try stopping them in case they were contributing to some other symptoms, I just stopped that day without any issue (even though the consultant said it might be "worse than quitting heroin").
On the other hand, I tried _one_ cigarette at university, and spend the next day or so with a near constant thought in the back of my head - "I should get some more cigarettes".