Spam filter is an interesting choice of motivating example, since usually it is your users labeling the data, rather than something that happens during the R&D process. You could try to use active learning but I'm not sure the users would like that product experience.