But they are diametrically opposed. It's not puritanical that I don't do drugs or that you're monogamous. What's puritanical is trying to impose those personal choices upon others.

It would be equally illiberal to mandate that everyone do drugs or be polygamous. The illiberality is the imposition itself, not the quality of the imposition.