I've always been surprised the CPC model works at all (and looking at Google it works like hell!), and this kind of reasoning (regardless of its ethical considerations) is one of the reasons.
The only people who are hurt are the legitimate publishers. When you click an ad you aren't interested in, you effectively lower the CPC for all publishers. This helps the site you're on, but hurts all the rest who show ads of the same category.