You aren't being force to interact with the FSF.
Alternatively, could you explain how you differentiate "we will not work with these people" and "forcing someone out of the community"? Like if enough people refuse to work with someone, doesn't that functionally force them out of the community?
I see two demands: the FSF board step down, and Stallman remove himself from any leadership positions. This is functionally the same request as last time, to keep RMS out of leadership. I'm unaware of any significant backlash to him having been a speaker at LibreConf, so clearly he could still still be a community member.
Outsiders are asking the board to resign over a hiring decision.
Edit: Wow the brigading in here is worse than Reddit at US sunrise.
It's open source software written by volunteers, though. How many people are actually forced to work with RMS?