Personally, my response to that is: "We can't live without harming anything else. I draw my line at plants, you draw your line at non-human animals, cannibals draw their line at human animals. We have different philosophies and preferences".
(And a Simpson's quote comes to mind: "Ha! I'm a level 5 vegan. I don't eat anything that casts a shadow")
It's a much simpler, broader, different intelligence and not much embodied in the part destroyed for food. In many cases consumption of the plant (fruit) is the plant's method of reproducing and sustaining life.