Comment by jacoblyles | Hacker News Reader

Comment by jacoblyles | Hacker News Reader

Parent Full thread

As a follow-up, I don't think it would be unreasonable for climate scientists operating on public funds to be required to take the following steps:

1) Archive a copy of all raw data series used in publications with a public database

2) Archive a copy of the final processed data series used in publications with the same database.

3) Archive a copy of the source code used to produce figures and tables with the same database.

The default stance for publicly-funded science should be openness, not secrecy.

Hell, these would be good steps for any serious journal to take, on any subject, regardless of federal funding.

edit: Example of the pathology we need to fight:

>>"We should be able to conduct our scientific research without constant fear of an "audit" by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues. In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science. I am unwilling to submit to this McCarthy-style investigation of my scientific research. As you know, I have refused to send McIntyre the "derived" model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully. I will be consulting LLNL's Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre."

I hope that this entire debate results in a new set of ethical standards for publicly-funded research. At the minimum I would support the open and free release of all unaltered data, a "Chinese wall" between researchers and activists (of whatever breed), and the requirement for all science based on "best guess" to say so in bold print, much like the language we have when investing with a prospectus. The standard of proof being used should be available in clear understandable language.

I fear that by the time the political debate ends, a lot of good research is going to become collateral damage, which would be an awful result.

Daniel, you say: "At the minimum I would support the open and free release of all unaltered data."

If i may extract quote from the message you replied to: As you know, I have refused to send McIntyre the "derived" model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully. I will be consulting LLNL's Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre." (emphasis added by me)

I do agree with you that transparency is the litmus test of good science. At the same time, catering to that can lay you open to abuses from, well, trolls.

As an analogy, suppose I had a bee in my bonnet about Linux security and I kept demanding that Linus send me copies of his binaries every time he did a new commit. He would rightly tell me to get the sources and build it myself and leave him alone. and in turn I could post lonely rants along the lines of 'why won't Linus Torvalds come clean about Linux security'. It would be quite meaningless, but it would sucker in some people. If I was sufficiently clever I could probably get quoted by someone at Microsoft or in the BSD camp.

The parallel I am drawing here is to the public datasets and methods laid out in published, peer-reviewed papers. If McIntyre is so sure that either the methods or the datasets are flawed, why not follow the established practice of writing to the journal and challenging the paper, or even submitting his own analysis of the public data and explaining how it improves on existing analyses? Instead he posts in some minor journal which is not part of the scientific corpus, not a hard science journal, and has been widely accused of lax publication criteria.

One of the things that really bothers me about the skeptical crowd (as opposed to individual skeptics) is that they employ a lot of same kind of arguments as the proponents of 'intelligent design' do, claiming there's an ivory-tower conspiracy that silences all dissenting views and shuts them out of publication. Any time you have a bunch of people going 'OMG teh conspiracy', it's time to whip out the old bullshit detector...and all too often, comparing their claims with the published literature sends the BS detector way into the red.

Nor will I provide McIntyre with computer programs,

Sad fact: most journal articles do not include all the details necessary to reproduce their results. There are usually all sorts of heuristics and tricks used which are not interesting enough to include in the paper. Typically, the experts in the field all know each other and share source code/tricks in private conversation.

As for the "ivory-tower conspiracy", it apparently isn't bullshit. As these emails reveal, there is a conspiracy to make it difficult for Steve McIntyre to examine published scientific results. There is also a conspiracy (revealed in other emails) to oust climate skeptics from scientific societies. It isn't paranoia if they are really out to get you.

why not follow the established practice of writing to the journal and challenging the paper, or even submitting his own analysis of the public data and explaining how it improves on existing analyses?

It looks like, from the emails, (and I hate to prejudge) that "submitting his own analysis" is not going to be an option for him.

Which leads me to the discriminating factor -- if publicly available data and publicly available processes allow an independent to process the data and reach different conclusions and be published that's fine. But that's not what is happening here.

There is no mystery about what happens between Linus' source code and the compiled executable. It's imminently transparent and reproducible. Science should be exactly the same way.

I also agree with having alarm bells ring whenever you hear "OMG the conspiracy!". I would humbly add, however, that groupthink is a real and prevalent problem -- not just among academics but everybody. If your groupthink reaches a point where you view that it's "us against them" and that your job in academia is to defend a pre-established position? I'm thinking you should be fired and never work in that field again. It's critical among scientists more than anybody else to have an open mind and a large dose of humility. The entire idea of science is that what you're working on will be replaced or fine-tuned later on down the road. Everybody is going to be wrong to some degree. It's part of the job. Big egos are not such a great thing.

I don't have a lot of tolerance for taking public money and playing politics with it, even if you are proven correct in the long run. One man's troll is another man's skeptic, and it's your job as an honest scientist to deal with them. Good scientists should first and foremost always be open about data and methods. Good grief, it's 2009, publishing all of this on a wiki somewhere is trivial.

"Why not ... [submit] his own analysis of the public data ... explaining how it improves on existing analyses?"

This is the meat of the matter. Scientists write letters to journals all the time saying that they think an author overstretched. Those letters, in and of themselves, do not negate the previous research. Scientists also write original papers showing that a previous hypothesis is invalid. Those papers, if they are well written, are easy to publish, especially if they make the journal more interesting.

Actually, I what you describe would be a useful requirement for all journal publications, not just in climate.

This is useful not just for the purposes of auditing, but also for the purpose of scientific progress. I would be able to build on other people's work much faster if I could download and modify their code rather than just rebuilding it myself.

There is a saying in the scientific community..."if you want to make sure that your competitor doesn't get any work done for the next two years, give him your code!".

In all seriousness, though, a lot of labs view their in-house codes as a competitive advantage. You may not like it, but science has become commercialized, the labs/unis are all competing about the same grants, which creates a competitive enviroment.

Reply on news.ycombinator.com