Hacker News Reader

Comment by matt4077 | Hacker News Reader

Who would have thought a story about reading thousands of pages of archived documents could be so riveting?

For anyone interesting in how this task has changed, there are similar meta-stories about, as but one example, the process of analyzing the Panama Papers. The investigative collaboration sharing access to them digitized millions of pages, and ran (among other things) named entity recognition and clustering / graph discovery algorithms on them to find the signal in that particular heap of noise.

If you’re looking for a project exploring / learning some new technology, I can only encourage you to try to find an angle that explores some public data or document set. That could yield something far more interesting than yet another RSS or HN client.

View on HN

Sure, but how do you find a project like that to enter at a useful stage? I feel like with the Panama Papers, any semi-public plea for help sorting through that data would have thwarted the journalists

The CIA discloses sets of documents from time to time as do other government agencies. The CIA recently (past few years) released a huge collection of PDFs regarding their investigation into psychic phenomenon. I only read a few documents at random of these (bizarrely the CIA seemed or seems to have some success with psychics) but I'd be interested if someone could synthesize them into something meaningful or explain exactly what the CIA was doing, to what extent they believe in psychics, and what evidence they really have.

https://www.cia.gov/library/readingroom/collection/stargate

In short, the Russians had a project to transmit psychic messages on submarines. This was of vital military importance if it would have worked. So the US tried that also, and then they also tried to use LSD (for interrogation, and other drugs) and subconscious hallucinogenic orders (for agents). This worked better, but of course was also shut down (MKULTRA).

They still rely on the equally unscientific lie detector tests, which works even worse than Scientology methods. But very similar.

Data.gov, the public datasets on BigQuery, thebInterhet Archive, etc. The Twitter API is also relatively open, so is Flickr. There’s a lot of data to play around with, even if not all is as scandalous as the Panama Papers (but who knows...)

Reply on news.ycombinator.com