Wait, is that true?

From the article, it does seem that the training and test sets were both generating from a dubious data collection process: scraping URLs they knew were fake/satire/real and then applying a label based upon the domain.

But, they had more features than just "source" in their model. so while it's possible their data collection method means that their model is way over-fit and basically just tests of proxies of "source", it's not prime facie obvious that this is the case...? Or am I missing something?