Libertarian here! If you don't really care if you step on a few toes to make the world better overall...
Then we should still be on the same side of this issue. I mean say we hate the idea of sex work overall, and there are real problems we can point to that it causes. So we make it illegal. Now we have two problems. It certainly won't stop prostitution. And it's the prohibition that creates the pimps, makes sex workers unable to report crimes to the police, fosters the most unsafe conditions, increases the spread of diseases, creates corruption.
No one should support this prohibition.