But the reality is that both the career advancement and funding for further experiments is almost entirely based on the number of publications in prestigious journals.
And effort put into collecting data is not by itself publishable: you must add analysis, and very often analysis that successfully confirms a new hypothesis or invalidates previous beliefs to publish. Journals prefer to publish exciting results, and "we ran experiment X, and found no significant effects" is usually rejected.
Fixing this is desperately needed, but so far nobody has come up with a solution that would actually fix this.
A middle ground would be to require data publication after a certain time period. For example, NASA and ESA release original data 6-12 months after collection. Releasing original data together with the paper also works.
Wouldn't rewarding people for `cites' of their dataset be a way to fix this?
Ie if someone uses your data, they have to cite you, and that's just as good as someone citing an analysis paper.
Edit: I'd love to actually, but perhaps I'll wait for the post scarcity, star trek economy to become mainstream.
Scientists have to eat and pay the bills, too. Unless you're really rich, it's tough to do much serious science in your spare time on weekends, paying for equipment from your own pocket. So the idea is that a scientist gets a job that pays him/her to do science. The system rewards scientists who advance knowledge and penalizes those who do not.
How do we determine whom to reward? It is extremely difficult to evaluate most scientists' work unless you are an expert in the same field. So we let peer reviewers evaluate the work, and reward scientists for publications in prestigious peer-reviewed journals.
The problem is that these incentives work against some kinds of sharing.
It's not all bad. A scientist whose work never sees the light of day is fired; no publications means no job. But sharing of ideas apart from those that are necessarily published is something the system tends to punish.
What to do about it? It's easy to propose some radical revision of the system. But we still have to deal with the question of how to evaluate the work of someone whose ideas can only really be understood after a decade of specialized study.
An optimum is often found that favors sharing as little as possible but often "as little as possible" is still quite a lot. Also, how can people label this as unethical when people would not even have gathered the data if they knew beforehand someone would force them into sharing, diminishing the data's return on investment?
I'm also pretty sure the sharing of a complete data set will get you a lot of citations, so there is a financial incentive next to the moral one.
But many clinical trial researchers want to withhold the data even after publication. This is fundamentally at odds with how science works. It means no one can independently evaluate the results.
Edit: no, on closer reading, I am incorrect. My bad.
Or shame them academically. You'd be surprised how far that goes.
It would be really worthwhile for the International Committee of Medical Journal Editors to hear your story, as they are proposing that authors should be allowed to withhold data for six months after publication. See my comment elsewhere in this thread: https://news.ycombinator.com/item?id=11019760
If you have access to PubMed Commons, I would post a comment warning others that the authors refused to make the data available despite promising to do so in the paper, and that people should treat the results with caution.