If you used HSL, you possibly could match on the hue while being more lax on the brightness and saturation—so a pastel-pink color could still match to a red image. Not sure if the database can use different weights for positions in a vector (though with pre-processing S and L could be collapsed to narrower ranges, so they match more freely afterwards). Just need to make sure that white and black aren't matched to random hue values.
Though, I guess if you use only the distance to fixed chosen high-saturation values then it's about the same thing.
Except it seems that with just RGB distance, a semi-saturated cyan or yellow color might be counted as closer to ‘pure green’ than dark-green. Something like (0,255,180) vs (0,70,0).
Also:
> values('(${ color })'
2020 and still no placeholders.