Well, I think you could say that it is a reason, but I don't think you can say it is the reason.
There are still a lot of people in the United States who believe that doing these illegal drugs is morally wrong, and that making them legal would increase, not decrease the damage.
I'd bet that this is driving the resistance to decriminalization/legalization much more than people trying to keep their jobs (and no, I'm not denying that there are people who resist it for that reason).
Depending on the drug you can
* Become unpredictable and therefore dangerous to your surroundings
* Become aggressive and/or violent
* Decreased cognitive abilities can make you susceptible to bad decisions
* Slurred speech and thinking can make you unable to help friends, family and others should they need you to.
That's just off the top of my head, please note that some of the above also applies to legal drugs.
Whether ingesting it makes you a danger to others due to intoxication or other loss of reasoning ability and whether ingesting it causes an addition that leads to the loss of ability to support yourself are likely to be two of the biggest concerns. Note in particular that the loss of ability to support oneself, combined with addiction, is known to drive other criminal behavior (muggings, theft, etc.) to support the habit.
This, incidentally, would be why more people are willing to be flexible with pot where there are few such dangers and less flexible with things like meth.
I do agree there are many who will object on these moral grounds though.