I feel like k-means in in Lab color space would lend itself well here too. Either by picking a large k that you then narrow down by picking the highest contrast colors, or instead by running k-means multiple times with different k's until you get an error threshold that is tolerable and results that have a high enough contrast. K-means is also a pretty easy algorithm to understand/implement.