Do you admit this is a problem, but you just don't see a solution compatible with the 1st amendment's freedom of association? Do you support the Civil Rights Act's restriction of freedom based on race and/or think it is constitutionally valid? Why would the same logic not hold for restricting companies such as Twitter then?
[1] Spare me rebuttals of "if they really wanted to communicate, they could do so by carrier pigeon!" - 99% of users won't go through with such effort, or even know who they are being herded away from. The remaining motivated 1% is too small to have any political power, and so the censor wins.