The extent to which the lack of universal open access is impeding human progress seems impossible to measure, but this gives us a tiny hint.
A more speculative hypothesis: this relates to the general "flattening" of the literature, as people relate to it more via search and less via subscriptions. People no longer really read journals as periodicals, or even use a specific journal's index as a search aid, but rather search entire archives for things. So it now matters less not only where something was published, but also when it was published.
http://researcher.watson.ibm.com/researcher/files/us-bennetc...
This is quantum physics. Look at where it was published. Proceedings for a CPSC conference in Bangalore. Bangalore! This paper started an entire field that entire departments at multiple universities are devoted to now. If it hadn't been so radical at the time it would have made it into a high impact journal. Instead, IEEE conference proceedings, Bangalore.
I went through grad-school in this field. This paper started it all so, needless to say, I wanted to see an original copy of it. I never found one. These conference proceedings were not widely distributed. People passed around copies of copies of copies and there were, of course, reprints in other publications. I had to borrow a book that included this paper as a chapter from my supervisor, and that wasn't long ago. The only reason I was able to read this field-creating paper was because, even though the peer-reviewed journals that should have jumped at the chance to publish this work balked, other people recognized something good in it. Today, a pdf of the original is linked to by Wikipedia.
How much bold, radical, totally unpublishable (in any respectable journal) work is out there waiting to be dusted off and used to start new fields of study? BB84 could have slipped through the cracks. It's a safe bet that other such work actually did go unnoticed. We can only stand on the backs of those who came before us if we can access their work. I don't know if what we're experiencing today is an information revolution, but I'm willing to bet, in a few decades time, we'll be able to look back on the march of scientific progress and see a huge boost in research productivity when our ability to distribute and search through existing research started to catch up to the pace of its production.
One of the key measures in chemical similarity is the "Tanimoto", better known as the Jaccard similarity. Quoting from http://en.wikipedia.org/wiki/Jaccard_index :
> Various forms of functions described as Tanimoto similarity and Tanimoto distance occur in the literature and on the Internet. Most of these are synonyms for Jaccard similarity and Jaccard distance, but some are mathematically different. Many sources[3] cite an unavailable IBM Technical Report[4] as the seminal reference.
Huh! Worldcat says that a few copies are available. Now I want to see if I can get a copy. :)
I recently had to talk to a library archivist to track down what turns out to be a key paper which describes a practical topological encoding of a molecule for a computer. The author published it as a corporate white paper. If you look at the publications from the 1950s and 1960s you'll see people who cite the whitepaper, but as it wasn't part of the standard scientific literature, it wasn't saved, and no one has referenced that work for decades.
As it happens, he's also the person who coined "information retrieval", and was one of the influential founders of IR. Thus, his papers were archived, and still accessible.
One of the interesting things about hard-to-find papers like this is to see how the citations change over time, as typos get introduced and propagate.
"How much bold, radical, totally unpublishable (in any respectable journal) work is out there waiting to be dusted off and used to start new fields of study?"
No doubt quite a bit. I'm always reminded of the FFT, which was rediscovered in 1965 by Cooley and Tukey in the 1950s. It was first invented by Gauss, 160 years earlier, but not well known, "being published only posthumously and in neo-Latin".
As a culture, is it better to investigate more of the history, or spend time learning how to create this sort of thing?
This should make people understand, that hiding knowledge away in obnoxious expensive science publications is one of the biggest hindrances to innovation today.
Science knowledge (when it is the truth) never grows old. Or would you think one moment, that the wheel is an invention so old, that we don't need it any more?
I'm still not sure how we get away with publishing items behind paywalls when a large percentage of the funding comes from the NSF/tax dollars. As an undergraduate, I was co-authored on two papers in fairly prestigious astronomical journals. The irony? I can't link my papers to potential employers or interested parties through the journal - I have to upload to arXiv to show people.