I'm not sure that the association is unfair. There are a lot of honest people out there plying the trade and doing good work in the SEO space, but the same could be said for email marketing. For every good guy, there's enough spammers out there that even now, guys like patio11 have to spend a decent amount of time spelling out the difference between legitimate email marketing and spamming. SEO strikes me the same way: lots of people doing good work and being drowned out by the negative press of snake oil salesmen, con men and link sellers.
It's unfair to the people doing the good work... but I don't think it's unfair that people free associate "email marketer" with "spammer" and "SEO Guy" with "Hucksters, Liars and Criminals". Too many run-ins with the Black Hat SEO crowd have made me wary and I'd be shocked if I was the only one.