I've used the embeddings (obtained by lopping off the final densely connected layers leading up to a classifier) from Alexnet and Imagenet, as distributed by Pytorch to try and cluster samples of art with HDBSCAN and didn't get great results. The graph generated by Mapper also doesn't make much sense.

Maybe there's a whole literature on this subject, but I'm way too thinly spread among interests and would love to know if there's something that's already done with standard parts to address this problem.