The poverty is quite related to the costs, and I think that's where I challenge your take.
That said, the most compelling take i have heard in the first half of 'In the realm of hungry ghosts' by Gabor Mate`. He points out that we need to consider the fact that some people may be much more predisposed to opioid addiction, by either default wiring or by life experiences. In this case, is the freedom of the majority of folks to take what they want worth the risk to those who would suffer from it? I don't believe he supports the criminalization of the drug use, but he makes a good point that we need to think about this reality of different-risks.
The reason why I come back to legalization is that I cannot tolerate the fact we have such massive funding sources for the cartels (and police) in the illegal trade. It is deeply compelling to me that we would be better off without funding that society destroying business.