I feel like it would only be right for this kind of project to not "stoop" to background-removal, but instead do something more ridiclous by throwing even more ML at the problem:

1. classify the image

2. use word-net to figure out what "container" word is most closely associated with the image's classification

3. search the container word, and pull out its dominant colors

4. return the colors that are in the image but not in its classificatory container. Unless that set is empty (e.g. a polar bear in snow), in which case return the colors you'd have given without all these extra steps ;)