Broadly agree. Freedom of speech definitely means that people may not want to associate with you any longer.
But 'cancel culture' wouldn't be an issue if that was all - if individual people simply chose to stop associating with the wrong-thinker. Not even if a lot of people did at once.
The problem is that when those people are not individual fellow citizens, but the "Trust & Safety Team" or the "Anti-Evil Team" at $Megacorporation, they don't just stop doing business with you - they can effectively cut you off from most of the rest of the world, whether or not the rest of the world wants it.
Simply put, the problem is that they have too much power. They have the same freedom of association as private citizens, but their ability to punish people is incomparably greater and isn't subject to the same restraint as the government's.