Hacker News Reader

Comment by spikels | Hacker News Reader

Because Hillary wants to make it hard to analyze them. Drug companies used the same technique to hide their relationships with doctors.

https://www.propublica.org/nerds/item/doc-dollars-guides-col...

View on HN

I was the original programmer for that project...while I didn't enjoy spending part of my late 20s learning about PDF to text conversions...I wouldn't necessarily attribute their use of PDF to devious motives. It's easy to forget sometimes that much of the world still operates on paper...and electronic documents that translate directly to paper have some value. These financial records were disclosed as part of legal settlements, and they were also used by hospital compliance officers...While obviously just sending out CSVs or Excel sheets would be great, it adds a possible breakage point between the company and its audience...i.e. hospital officers complaining that the spreadsheet looks funky on their version of Excel or whatever the hell they're using. The thing that makes PDFs so frustrating to parse is what makes them so "secure" and stable to end-users. That's why I advise my students to not just send URLs and web-clips to potential employers, but to have PDF versions of their resumes and work because inevitably, the "boss" wants a printout on his desk...and you don't want his office's shitty printer (and/or whatever IE browser rendered the HTML) to leave a bad first impression.

And let's face it, if you are an end-user who is not a programmer, a PDF is very convenient. You can even do Ctrl-F to look for things...and if you aren't a programmer, that's all the search-power you need to do your compliance work. After the investigation was published, I fielded a lot of questions from auditors who were convinced I had built some amazing proprietary search algorithm...such people just don't know the gap between doing Ctrl-F versus "SELECT * FROM doctors where LAST_NAME like ..." :)

There were a couple of companies who arguably went out of their way to obfuscate their records. They had built websites that consisted of Flash widgets designed to look and act like a (crappy) HTML table...except with Flash, you can prevent the user from select/copy/etc. Unfortunately, the developers of those widgets never told the companies that their Flash containers operated by making XHR requests to external XML/JSON files...so these companies ended up being the easiest to gather the data from.

Handling PDFs and OCR is always troublesome.

But in this case, if I understand you correctly, the original source documents were PDF? That they were distributed as PDF for all the reasons you state?

That doesn't apply in this case - the original documents were not PDF, and had no reason to be PDF except to make it more difficult.

Fortunately, the PDFs already contain the plain text as metadata. I believe they are what are known as a searchable image PDFs.

The code posted here isn't doing any OCR, but whatever generated the PDFs (Acrobat?) might have.

How did they get the plain text as metadata? Was the scanning equipment doing OCR and setting that?

I believe the U.S. agencies use ABBYY FineReader, which does a pretty good job with OCR and text resolution. The U.S. Senate used it when releasing the CIA torture docs awhile ago:

http://www.nytimes.com/interactive/2014/12/09/world/cia-tort...

According to news reports Hillary handed over the emails to the Dept. of State as hard copies. These were then processed by them into PDFs including doing OCR.

http://money.cnn.com/2015/03/11/technology/security/hillary-...

Edit: Hilarious part was where State says this is not a problem because "it would have to print Clinton's emails in the normal review process."

She also handed over Outlook database files on a thumb drive to her lawyer, so it's hard to argue that this whole process was somehow necessary.

Sorry, I accidentally downvoted you when I meant to upvote you! Those arrows, ever so tiny...

Reply on news.ycombinator.com