I'm no color expert but I'm pretty sure color distance in RGB space is a pretty poor metric compared to other color models (at least some of them). Looking at an RGB color cube I see big splashes of near-identical colors while the entire spectrum of oranges is hard to even find.