Here's a blog post about this issue that you should find interesting. http://www.psychologytoday.com/blog/tinted-lenses/200912/scr...
My personal point of view is that I do not want to cause any suffering when I can avoid doing so (i.e. when my survival does not depend on it). Only a conscious being can suffer, and, as far as I know, plants are not conscious. There are organisms of all levels of complexity between single-cell organisms and homo sapiens, and no-one knows for sure where true consciousness begins, but the line between things and beings needs to be drawn somewhere. I have personally drawn the line between plants and animals, just to be on the safe side. It may be too far, and there's a risk that I'm not using some things for fear of causing suffering to a conscious being, but I'm OK with that.