I was not sure about this, but if this is the case then their conclusion seems to be unrelated to their findings.
I just looked into the actual data set[1], and analyzed a few papers from it, e.g. [2] the first paper in scopus_nature.csv. They're definitely available on Sci-Hub at least at the moment [3], and there's no reason to believe they weren't available back then.
Also, here's what they say about the dataset, emphasis mine:
> As a quality control, we performed a random sampling of all the articles retrieved, excluding those already present in the first data set. As in this second data set, the number of Sci-hub downloads is precisely equal to zero, we regard it as a control group (nC = 4,015) from which we are going to estimate comparisons for our experimental group (nE = 4,646).
Downloads equal to zero != not available.
[2] https://www.scopus.com/inward/record.url?eid=2-s2.0-85016141...
I'm wondering about this too. What could be the possible other reason for the papers not being downloaded? Boring titles? Preprints easily available? Open access?
> and there's no reason to believe they weren't available back then.
Is it possible that the articles really were not available, as they used the data from sep-2015 to feb-2016, early version of SciHub?
That's within the realm of possibilities but sure doesn't sound like the case the way they described their methodology. That would have required a catalog of available papers on Sci-Hub too, but the dataset they used only contained download logs, so they couldn't have known whether a zero-download paper was available or not (unless I misunderstood what the first dataset is about).
Of course, that is hardly a surprising result. I think most authors have read most papers they cite (although things like reviewer/editor shenanigans could make this less than 100%).
Of course, such a pedestrian result does not grab attention.