No doubt, opt-out should not suffer these problems. Is it SPAM after that point? Maybe. I can see at that point the reputation system wins.
`it is spam to everybody`, maybe, but Google is trusting the user to categorize SPAM, which can have some unwanted consequences.