When assembling the data set, anything that was clearly 404'ing or unresponsive was pruned out (one or two might have slipped by, do let us know if that's the case). Fresh data is of course harder to gauge.
Frankly, I was surprised -- and impressed -- that the number was this high. Great stuff.