This might sound glib, but it is a big part of the kernel of Constitutional scholarship; Ely's _Democracy and Distrust_ goes into detail about substantive versus procedural due process and how the core of the effort is to "preserve the channels of democracy". In other words, Constitutional Law's most important job is to be a referee, to ensure that the political process isn't abused to lock groups of people out and, in doing so, to preserve injustices.

Restrictions that obstruct the channels of democracy should have an especially high burden of necessity to clear.

Americans lost rights to buy marijuana in 1937. Winning it back by voting turned out not that easy in next 70 or so years.

And, btw, if you try to still use you right to own marijuana, you become a felon, which - surprise! - means you lose rights to vote. See how nicely it works out?