Oh I see I assumed the GP was talking about after such a dataset I agree you likely couldn't bootstrap without a lot of data but couldn't you use more classic approaches to extract data and then train a model (obviously once you know the extraction process is fool proof?)

But this is not the only way you can deploy machine learning techniques. You could deploy unsupervised learning to first learn what visual features are common in such images, and then successively refine until you end up with a set of images containing very rare phenomena. That could give human evaluators then a considerable head start.

In fact, this idea seems so obvious that it must have been tried already…