Hacker News Reader

Comment by tlrobinson | Hacker News Reader

It's interesting this person can program, but chose to manually count 27,740 Skittles instead of automate it in some way.

It feels like an almost-trivial computer vision problem.

View on HN

Since no one seemed believed it's almost trivial, I went ahead and implemented it:

https://imgur.com/t7ivg0V

It took about an hour of Googling OpenCV questions. Also, I've never actually used OpenCV before, and don't use Python regularly.

I haven't tested it against the full set of images. Even if it's not perfect you could at least use this to narrow down ones that are within some threshold, then manually verify they were tagged correctly.

I'll post the code shortly.

...but it's almost trivial to count them as you go along, in addition to laying them out.

Have you verified the author's claim that this is the only match in the whole data set?

It's much quicker to dump the bag of Skittles out and snap a photo than to arrange them and count them.

My program isn't perfect, but I only spent a couple hours on it. I think it could be improved, and better lightning when taking the photos would help too.

However, I was curious how well it did so I compared it to the author's results. My program misclassifies approximately 80 out of 27,740 Skittles.

I searched for matches and it did indeed find the same match the author found:

    (0, './skittles/images/334.jpg', './skittles/images/464.jpg')
    (1, './skittles/images/103.jpg', './skittles/images/384.jpg')
    (1, './skittles/images/111.jpg', './skittles/images/134.jpg')
    (1, './skittles/images/118.jpg', './skittles/images/353.jpg')
    (1, './skittles/images/139.jpg', './skittles/images/168.jpg')
    (1, './skittles/images/152.jpg', './skittles/images/281.jpg')
    (1, './skittles/images/158.jpg', './skittles/images/244.jpg')
    (1, './skittles/images/198.jpg', './skittles/images/255.jpg')
    (1, './skittles/images/198.jpg', './skittles/images/334.jpg')
    (1, './skittles/images/198.jpg', './skittles/images/464.jpg')
    (1, './skittles/images/201.jpg', './skittles/images/338.jpg')
    ...etc

(first row is identical, subsequent rows are one different)

Since there are some errors, I did get lucky. However, if it's good enough, you could also manually verify ones that are close.

Anyway, personally I'd rather write code than manually count 27,740 Skittles, but I'm grateful the author did this because it's an interesting dataset to work with.

Author of the article here; I suppose I should confess that I did consider automating the counting, which did seem like an interesting problem... but I rejected the idea, for what I think is good reason: the sorting was the time-consuming part. If we skip the sorting, and depend on the code to get the counting right, and we assume that we are at worst off by one-- or more realistically, off by two, to account for the questionably-identifiable chunks of paste and such-- then how often do we need to go back for manual verification?

With hindsight (although up-front simulation bore this out as well), we see that we would have to go back and manually verify anywhere from 8% (off-by-one) to 25% (off-by-two); that's every fourth pack. Now consider how much more unpleasant that manual counting is, since the Skittles are haphazardly un-arranged in the image. In short, I'm unconvinced that an automated-- while similarly accurate-- accounting would be that much more efficient.

> the sorting was the time-consuming part

I'm not sure I understand why sorting is necessary.

> Now consider how much more unpleasant that manual counting is, since the Skittles are haphazardly un-arranged in the image.

It's actually very easy: my program outputs images annotated with each recognized Skittle circled with the guessed color: https://imgur.com/a/jlPWXRf You don't need to manually count, just make sure all the circles are the correct colors.

I'm also pretty confident you could improve the accuracy quite a bit by improving the lighting when taking the photos, and/or incorporating something more sophisticated like a neural network (probably trained with Skittles from several bags, separated by color+rejects, with many photos in different random arrangements)

> You don't need to manually count, just make sure all the circles are the correct colors.

I guess this takes me a while, and seems significantly more error-prone to me than the corresponding re-count when they are sorted. Granted, it's pretty easy when the Skittles are sorted as they are in these images. But how quickly can you scan this image: https://imgur.com/a/KpddGdH and check whether there are any errors? And how confident are you in that visual check?

You are right that this approach may be good enough to find a duplicate, which was the primary objective of the experiment. But I had hoped that this might also serve as a useful dataset for future student exercises, in probability, or even in just this sort of computer vision project... but I wanted to have accurate ground truth, so to speak. Inspecting your spreadsheet, it looks like this algorithm is still less than 95% accurate, even if we only evaluate the "clean" images with Uncounted=0.

Any update on posting the code?

https://github.com/tlrobinson/skittles.py

It's definitely not perfect but could probably be improved a bit.

There are a few where strawberry and grape or strawberry and lemon are swapped. I think better lighting would help.

Nice!

> I think better lighting would help.

That also demonstrates the difference between having the possibility to optimize input until the output matches the known values and having to process "real life" images made under different conditions.

The later is certainly more interesting: e.g. I dream of having an OCR that is "always right" no matter how bad the image is.

Controlled lighting is extremely helpful in computer vision and key to the operation of a lot of industrial uses of it. From a system design point of view it makes a lot of sense, but obviously it's not possible in many applications (and researchers obviously want to go after the toughest problems, and would rather mostly solve a hard problem than entirely solve an easier one).

Thank you, forked! I'll be learning from this.

FWIW I basically just cobbled together Stack Overflow answers :)

This is awesome.

What would it take to determine which brand (skittles, m&ms etc) and which sub-brand (tropical / peanut, etc) and how many calories are on the table?

In my head, you would determine the relative color difference of each piece to determine the brand, but I'm not 100% clear on how that kind of ai works.

Thanks. I have no idea if it's the "right" way to do this. It's the first computer vision problem I've done in a long time.

With better lighting you could probably distinguish between colors of different types of candy, assuming they're different enough. If you can determine the type of candy and know how many calories are in each piece it would be easy to count the calories, of course.

FYI all of your new comments are being marked as [dead] for some reason.

It's been done! A teenager built an M&M and Skittles sorting machine a few years ago. I expected this article to be based on that, but to my surprise, he counted them all manually.

https://hackaday.com/2017/02/06/mms-and-skittles-sorting-mac...

To the author's credit though, doing it manually probably made more sense in his situation. It would take a few months to build the auto-sorter, which would've had to be tweaked to handle the misshapen blobs. The collection of Skittles photos is impressive too, and hopefully will end up in a Maths classroom.

Hard to imagine that process would be trivial especially since as he noted in the article he had to have criteria on when to count a skittle such as if it were too small, appeared unedible and other such characteristics. But after completing the manual way would be interesting to running programs on it to see how close they are from the actual numbers.

computer vision is trivial? when did that happen?and how was it made trivial?

I've done a few computer vision projects lately and it's really amazing what you can get done with opencv for relatively clean data. In this case you're trying to identify circles and classify their color into one of several groups. This is pretty common, the main challenge is cleaning up the raw data so that the hough transform runs quickly and accurately, and the color cluster identifier is robust.

If you're pretty good with Tensorflow, you could do it with an object detector. I've built absurdly accurate object detectors using the TF Object Detector tutorial and just a little hand labelled data plus some good synthetic augmentation.

For this project's scope and scale it's probably not worth automating unless you're going to be repeatedly running the process.

He's saying detecting skittle colors is "trivial" in the field of computer vision, and he would be correct. A 50 line Python script with a popular classifier library would solve this problem with high accuracy. This is perhaps the easiest problem to solve with CV and would be a great first tutorial for such libraries.

http://opencv.org/

Counting the number of round colored objects on a white background is not a hard problem.

I think I've found a project for this evening...

Reply on news.ycombinator.com