Hacker News Reader

Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte. The per-bit cheapest disk right now is consumer Seagate 24TB(SI; 21.8TiB usable) at ~$500, or around $1200k for just the disks.

Doable if you had couple million dollars to burn. Cheaper than private jets new.

View on HN

Seems like the most practical path is some kind of distributed peer to peer contraption where you can allocate some storage and optionally participate in crawling.

I imagine if set some constraints you could get index size down quite a bit but I still suspect it'd be hard keeping up with content churn.

Edit: Looks like enwiki bz2 is coming in around 46Gi which isn't too bad considering the amount of content it contains.

Reply on news.ycombinator.com