IDK about "horrible" but https://www.sentencingproject.org/research/ puts it at 1.5 million peak. Thats not quite 1 percent of the US population but its also not that far off. And we can quibble about US sentencing rules but the BoP has a count[1] of felony categories and drugs make up less than half, and its even less in state prisons.

The reason the estimate feels high is likely that the HN crowd is progressively separated from less stable elements of society. The bullies picking on kids at recess don't generally make it onto the gifted / accelerated track, or your 4 year degree granting university. You're going to encounter fewer of them at upmarket shopping centers and corporate offices, if for no other reason, 1.5 million people are in prison.

[1]: https://www.bop.gov/about/statistics/statistics_inmate_offen...