Yep, you're right. It's difficult to differentiate greys that are important vs those that we don't care about. Our current solution works not too badly and gives us results that are more visually appealing, but I agree we could do better.

I'd like to try ranking greys lower somehow.. maybe some kind of weighting by saturation could help.

This was my first question too. In that picture, not any grey would match with the reds to give the feel of that image.. probably.. Maybe just order by saturation, and then limit to 5 colors?