The only question is what is the precise coefficient of friction on that slippery slope.
You can argue that wedding cakes should be baked for gay couples but that websites should not be hosted for nazis for reasons A, B, and C. But claiming that businesses have some sort of broad right to free association is not merely specious, but completely and obviously wrong.
Businesses do have a very broad right of freedom to serve who they please. The one exception is when historically many businesses have used that power to participate in oppression against specific downtrodden groups. So a business can happily decline to serve US congressmen, freemasons, and people who have thrown up on their floor previously. But they can't decline to serve black people as a group.
The specific groups who have been legally recognized for this are called protected classes, and it's pretty settled law in the US: https://en.wikipedia.org/wiki/Protected_group
Patreon, and other payment processors (like Stripe.com) are having their arms twisted from the highest levels by Visa and Mastercard.
Think for a moment if it would be ok with you if Visa and Mastercard were doing this because they wanted to silence people who advocated for women's rights. Would you still be so libertarian about it?
I think there's also a fundamental difference between an organization that bans people who increase the rights of marginalized groups and one that bans people that are looking to further marginalize people, so your analogy doesn't do anything for me.
It's not just Patreon. If anyone sets up a similar service, and does business with someone that Visa does not want, they will use their monopolistic power to shut you down.
Imagine if they were doing this about a cause you disagreed with. Imagine if our financial industry was run by Trump supporters, and they were slowly squeezing their critics out financially. This is bad.