What's the alternative? Does a free speech absolutist need to never make judgements on what he hears from someone's free speech? If I hear a person say he wants to murder me and my family, do I still need to invite him over for Thanksgiving dinner? Aren't I exerting some form of control of over him if I say you can't come into my home?
I think your view of the term "free speech absolutist" may make sense when analyzing the individual word meanings, but doesn't make sense as a phrase, and doesn't align with how self described free speech absolutists view themselves.