Hacker News Reader

Comment by pyre | Hacker News Reader

1. The Wikipedia estimate probably only includes text. All of the media is distributed separately, and was ~200GB as of a few years ago. I can only imagine it's grown since then. Some articles make more sense with images (e.g. a photo of an animal to go with the description).

2. Is the entirety of Twitter really useful as part of a Hitchhiker's Guide to the Galaxy?

View on HN

As a thought experiment, packing up the whole of a website is a good upper bound. If we were doing this for real, we could further aggressively trim, taking advantage of the fact that content utility tends to follow a power law rather than a uniform distribution. For instance, take stack overflow; plink everything with a negative point score. Plink everything with no replies and no votes. Plink everything that has a reply with more than, say, 5x the points (i.e., if there's a 100 point reply and a 4 point reply, plink the four-point reply). Plink entire sites/tags that are unlikely to be useful (no Python software on the mission? No Python interpreter shipped? Remove all Python questions). For Wikipedia, remove all information about localities that have either less than 5KB of text or 10,000 people. Spend some time hitting the "random" button on Wikipedia and you can get a sense of how much of Wikipedia does not need to be sent into space. Remove vast swathes of the catalog of each species on Earth... I doubt a Martian settler is going to have a compelling reason to suddenly look up a cricket particular to northern Utah. And so on.

You could probably cut all those estimates down by 90%, with very little loss observable to the users. It wouldn't be zero loss, but use the savings to stuff other useful stuff in there instead, such as redundant copies to ensure we don't lose anything.

Im making my own offline wiki (fun side project). EN wiki dump is ~10GB. 8-9GB if you preprocess XML dropping useless tags and wikimarkup

  falink = {b'Link FA|', b'Link GA|', b'link FA|', b'link GA|', b'Link FL|', b'link FL|'}

different language links/references, every article has a list of all the language version links (ex: [[af:April]] [[als:April]] ..)

  langcolon = {b'aa:', b'ab:', b'ace:', b'af:', b'ak:', b'aln:', b'als:', b'am:', ................-my:', b'zh-sg:', b'zh-tw:', b'zh-yue:', b'zu:'}

I have a feeling changing wikimarkup to something more size optimized would help even more. Swapping compression algorithm can squeeze additional 1-3GB of savings without sacrificing decompression speed.

All in all with a lot of tricks you can go from 10GB to 5GB. Doesnt feel worth it. Bigger problem is searching thru this data on a 2-4 core ~1GHz ARM processor with 1-2GB of available ram. Indexing titles is simple and fast, but quite useless. Indexing all useful words would eat at least couple of gigabytes. Searching for a simple combination of three words would take couple of seconds every time. Its not like you can haul Hadopi cluster in your pocket.

One post on Twitter: 140 bytes max. One HD frame on Netflix: 6220800 bytes max.

If throwing the near-totality of human cinematography on H2G2 is a no-brainer due to its manageably small size (a mere 24TB), then may as well throw Twitter on there as well.

Don't forget Unicode. 140 Characters =/= 140 bytes.

Don't forget compression too. And don't forget most tweets don't use close to the full available tweet data space. A billion tweets takes up a lot less than 140GB.

I'm just looking at rough comparisons. A single 1/24th of a second of Netflix video approximates 44,000 tweets' worth of data. Anticipating growth, let's assume a billion tweets per day (was half-billion last October). 171 days of tweets takes up the same space as the complete Netflix library.

It's probably easier to index and search Netflix than it is to index and search all of Twitter on a small device.

I'm probably not clear enough in the blog post: The intention is to use Twitter as a news source, i.e. crawl and index top URIs (which can be any type of news, blog and other content). The underlying idea is that URIs on tweets give a good sample of _all_ overall knowledge production per day.

There are also of course some benefits of having all the interaction happen between you and your device that I haven't talked about in the blog post, e.g. increased privacy (no data collection), lower latency (disk seek on a mobile or tablet SSD - 100 microseconds) - is roughly 1000 times lower than the latency of accessing 3G or 4G can be (up to hundreds of milliseconds)

They throw netflix in there, I guess the difference between 10 terabytes and 20 terabytes isn't going to stop the show.

To your second point, I totally agree that an offline internet isn't what I think of as far as a guide.

They separated it into text (20GB) and media (200GB).

But wiki is VERY wastefull when it comes to media. 2MB SVG file to represent 3 atoms worth of molecule? yep thats wikipedia for you.

Reply on news.ycombinator.com