Our three year saga to release 13M pages of CIA secrets
muckrock.com
muckrock.com
> The declaration also says that CIA cannot release these TIFF files in electronic form because they can be so easily altered by the mere act of a CIA FOIA analyst looking at them, and that the security measures they must take to remove this accidental metadata for an electronic release (involving editing each file separately by hand) would take 28 years and 1,200 CDs.
Aside from the issue of the CIA apparently not being able to develop (or download) a batch metadata editing tool, a common excuse by agencies for releasing data as PDFs or image files, instead of Excel/CSV, is because the latter is too easy to manipulate by the public:
http://www.nytimes.com/2010/04/13/business/13docpay.html
> Among the four leading drug companies making physician payment disclosures, Mr. Coukell said, Eli Lilly, which was the first to disclose, presents data as an Adobe Flash image, which he said was impossible to download or to sort. “They’ve gone out their way, I think, to present it as a Flash document,” Mr. Coukell said.
> Mr. Dunston said Obsidian had to retype all the Lilly data.
> Carole Puls, a Lilly spokeswoman, said the company purposely made its report impossible to download "to protect the integrity of the data." Lilly was concerned someone could change numbers and create a false report outside the company’s Web site, Ms. Puls said.
This could be easily solved by releasing all documents and a list of their hashes simultaneously.
The defense and intelligence communities hire from Tier 3 schools in the DC, Maryland and Virginia area.
People that - you imagine - would get laughed at for not being from Stanford and Ivy League at every VC pitch meeting. People you would think weren't the "best of the best" hired by all the tech companies that brag about doing so.
Of course, their engineers are just are educated and ambitious, I'm just pointing out how warped people's perception has become.
I mean, do you actually hear yourself? Really listen to yourself and describe what you hear.
Stupid and dangerous. You know these are the people who protect you from all manner of sabotage and manipulation from foreign nations right?
1. Screenshots of data can be manipulated
2. Digital signatures exist.
I like how they try to make it sound like it's a bunch of data. Sure, it isn't a small amount, but it's only roughly 840GB.
So it's more like
> It would take someone a few minutes to setup a batch meta data editing program (shit, or only an hour or two if one had to be made) and an external hard drive from office depot.
* Thinking things like astronomical data or other petabyte+ data sets.
I believe the UK signed up along with the US as part of the G8 to an Open Data Charter [1] in 2013. In theory it aims to ensure that all government data be published openly by default, but in practice, as far as I can work out it has no teeth. You can point government departments to it, tell them they should release data, ask why they wouldn't and just get stonewalled.
I particularly pushed hard to try and get data from speed cameras released (without personal details; just time, location, speed, etc.) I even offered to aggregate data, normalise it, run the hosting, etc all at my own cost. Nothing. As far as I can tell there's no one to go to who has the power and influence to make it happen.
[1] https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annexSeems like a good grad school research project.
Now imagine you are someone else tasked with writing something that captures "the substance" of those emails. You've basically been asked to capture "the substance" of someone's life over a 2-month period. The reason why you don't see many "deep" articles about the content of the emails is because John Podesta is someone who is interesting to the public for a specific role: Clinton campaign manager. So the contents of his emails are analyzed in the context of the presidential race, not for any other kind of deep contextual analysis.