By the website's definition, which is right on the main page, it is not wrong. The website uses the "average color across the approximately 25 image results".
And for that purpose this site works fantastically, and I would say it's a pretty hilarious idea.
For example searching for Dolphin doesn't return grey but instead a shade of blue, because most dolphin pictures are dolphins in water.
I think the website takes our expectations and instead shows us the reality. And I think it does a fantastic job.
What I would like to see is an option to maybe increase the number of images to average from, maybe to 50, 100, etc. I would also love to see the source released so I can this instance locally.