I'm not an expert in the topic, but my understanding is RGB is a poor color space for computing color difference. This could be why your mosaics end up so washed out. [1] suggests using a CIELAB color space [2].

Edit: Looking at the code more closely it looks like you were using Lab at one point but commented it out[3], so I'm guessing you're already aware of this.

1: https://stackoverflow.com/a/9019461/185171

2: https://en.wikipedia.org/wiki/CIELAB_color_space#CIELAB

3: https://github.com/worldveil/photomosaic/blob/bb720efda11383...