There are also legal drugs that have similar effects to things like Adderall. A lot of people don't know about them or use them because they haven't been researched as much and come in powder form.
At the very least I think that you shouldn't be put in prison for carrying pills without a prescription.
You might find this Vice article interesting. It talks about how Oxycotin is essentially legal heroin.
http://motherboard.vice.com/read/how-big-pharma-hooked-ameri...
I say we should take it slow when it comes to legalization. Perhaps start with things that are found in nature. Marijuana, psilocybin mushrooms, being legally able to harvest the opium from poppy seed plants, etc.