I'm gonna guess you've never been arrested, much less seen the inside of a jail cell. So it's not "just you," but you are expressing an existing stereotype of civic ignorance.
Pot being illegal causes uncountable amounts of suffering and disadvantage, and it's not going to be federally legalized (descheduled) until enough states pass their own legalization.