It could, and indeed there are people who hold this belief. They are called "Fruititarians" and only eat things which can be taken from the plant without killing or harming it, see:
https://en.wikipedia.org/wiki/FruitarianismI don't personally subscribe to this.