This article from Gizmodo does a good job of explaining the difficulties involved: http://www.gizmodo.com.au/2015/10/why-is-it-so-hard-for-robo...
Well, conceptually it sounds simple. But it is certainly a complex engineering problem. You are forgetting building conforming actuators and sensors, controlling these, image registration, the low margin for error, classification performance, real-time processing, etc. etc.
Applying the principle of capitalism, if it was simple to design and if a market and technology enables it, then it should already exist. In this case, the market obviously exists (e.g. farm worker demand in fruit industry), so if the solution doesn't already exist, then it is highly likely to be because the problem isn't simple to solve and/or the technology is inadequate.
For fruit sorting in particular there is the very difficult task of conveying the fruit without damaging it. The best sorter in the world won't help you if it mangles the fruit.
In my old lab, there are solutions to most of the common food sorting problems, not all of those solutions make it out to the field though.