And the total data set is less than a terabyte; seed a torrent somewhere for $20.
The user-pays S3 bucket also exists as a good thing but S3 is much more expensive than data needs to be.
I agree that it would be an extremely valuable course of action to seed a series of torrents, since a single torrent wouldn't work; it would have to be replaced every time a new paper was uploaded, fragmenting the swarm enough to render it useless. Also, they could surely use Fastly and permit spidering.
The collection was a lot smaller then. The whole thing has fit on one hard drive for a long time. And as far as interpreting my post as criticism goes, apply it to the last ten years only.
> This is the context in which the arXiv's hostile stance toward spidering was established.
They should have reconsidered it at some point.
> series of torrents
Sure. A torrent for each 500MB chunk they already collate, or yearly, or both.