Case in point: Monsanto. Their notorious Roundup-ready plants have been GM'd to be resistant to herbicides that will kill off all other plants.
I would cheerfully eat a plant that had just been GM'd to have such resistance. But I would be rather more concerned about it after it was used as designed: repeatedly sprayed with massive doses of toxic substances.
Even if they're completely correct about all traces of that gunk being gone by the time their plant winds up on my plate, you still get the herbicides and/or the breakdown products washed into the water table; you still get farmers skipping crop rotation and fallow seasons in favour of just growing the same crop over and over and fixing the perceived problems by dumping yet more fertiliser and yet more herbicides onto them.
And you still wind up encouraging the evolution of herbicide-resistant weeds requiring yet-more GM and yet-more poisons. You still get yet-more need for yet-more poisons of questionable safety being dumped into the food chain.
I've no problem with eating a plant that's had its genes tinkered with. But I'm totally against Roundup-ready crops and the like. And that's a perfectly rational, informed position to hold.