Hacker News Reader

I agree that code should be made public, but I couldn't disagree more with the last paragraph.

Even worse, I take offence to the sentiments expressed in it. It's time to set ethical standards?! Really? Are they so unethical? I would claim that scientists have, by and large, very high ethical standards. Considering that they themselves are being pushed more and more to market their research to get funding, the highest standard for our work remains truth, and nothing but the truth.

Elimination of scientists as activists is a non-statement. Assuming that scientists believe in their own work --a very common affliction-- what does it mean to be an activist? Just that they try to convince others of the validity of their work? Isn't it unethical _not_ to warn the world of the impeding doom your research has uncovered? And to be somewhat insistent if people do not want to hear it?

Don't confuse politicians, crooks and businessmen in the climate change debate with the scientists. While all pretend to use scientific arguments, very few do. The political question should be: does the risk of climate change justify the costs that might prevent it.

The risk of climate change shouldn't be political. That needs to be studied, rigorously. Al Gore may have closed the debate on whether climate change is happening, but to what extent is far from settled. Are X costs worth preventing Y damage, should be the political debate. We don't know Y yet.

"And the only reason it's coming out is because some people -- for whatever reason -- have a bug in their shorts about climate science"

On the other hand, scientists have spent decades fully aware that they'll have to build their own labs, obtain their own equipment, get their own animals if the work requires them, and follow the described procedures.

They can't use the other guy's mice or zebrafish or monkey. They probably can't use the other guy's lab. They might be able to use the same telescope for astronomy, but it's better to use a different telescope to control for some fluke of the original.

So given all that, it hardly seems a huge deal to not be able to use someone else's code, as long as you know what the code was supposed to be doing.

"So given all that, it hardly seems a huge deal to not be able to use someone else's code, as long as you know what the code was supposed to be doing."

But that's a vacuous statement. The only description of what the code is doing is the code. The description you get in a paper is the goal of the code, but goals aren't results. (Mistaking stated goals for results is a surprisingly common systematic error, programmers make it all the time when they believe the hype a project puts out before the project actually has any results. Once you start looking for it it's hard to see a day go by without someone doing this.) I can make a simple information theoretic argument that it is blindingly obvious that any significant code base will have many more bits in it than any journal article could, so it is literally mathematically impossible for a journal article to accurately describe the code. If goals were the same as results all of our jobs would look very different.

Moreover, the reason why you can't use the other guy's monkeys is purely physical. If they could use the other guy's monkeys, they would. They try as it is; the entire purpose of controlled breeding gene lines is to try to erase variations. If they could use guaranteed-atomically-identical lab hardware, they would. (The story of cold fusion would probably be very different if this were possible, for instance; instead of an effect that nobody could reliably replicate, there would be one set of results that everybody could replicate and probably rapidly explain. Note I'm not making a claim about cold fusion itself, just the way history turned out.) Don't elevate an accidental physical limitation to an essential component of science. Very few things hit the true ideal of science, and in general the closer you can get, the closer you should get.

"If they could use the other guy's monkeys, they would. "

No, they wouldn't, any more than a scientist would use only one monkey. Because animals vary. Hell, my Mom had two uteruses. If you don't look at different specimens, you don't know if what you observe is peculiar to the one specimen, or generally applicable.

Using atomically identical hardware would be a bad idea, because then you wouldn't know for sure if your results are replicated because the experiment was good, or because of some quirk in the apparatus.

I only say it is desirable to be able to replicate results. Nowhere do I say that science must consist ONLY of replicating results. It actually is a bit weird to me that so many people leap to this conclusion when it is obviously falsified by other existing sciences where results absolutely can be exactly replicated and yet scientists do not deliberately discard that ability.

Replication of results is merely a step in the process. Once you establish replicated results, you proceed from there. Maybe you observe that the results are irrelevant because your monkey is weird in some way, and you test it on another monkey to establish that point; others can then examine their copies and decide whether you've got a point. Maybe you build on the experiment with a standard test bed. Maybe you take apart the apparatus to demonstrate how a bit of impurity corrupted the results, and then everybody else can do the same disassembly. Maybe you shuffle in some new monkeys and hardware and run the test again to be sure. Not being able to precisely replicate results is a handicap that the various sciences proceeds through anyhow because it has no choice, not a desirable part of the process. Sure, there's a small amount of danger that you might overfit your results, but there's no guarantees anywhere; it's less than the danger that you face from non-replicable results. Not having replicability only throws away options, options that smart people could use to further their science, option that when missing can only slow progress down. That dumb people might misuse it really isn't a very interesting point.

And again, I reiterate that to the extent possible animal researchers do in fact try their best to get animals identical as possible, so I give you not only the theory I outline above, but the practice of biology as well, where carefully controlled standardized gene lines are used.

I work in a lab that does monkey research. What you say is more true about smaller animals like mice, where there are lots of variants, knockouts, etc.

Macaques don't really offer those kinds of options, at least not that I've ever heard of. You just hope they're healthy, disease-free, relatively smart, and are easy to get along with.

on standardized platforms

This would be horrifying. Maybe it could work in some fields, but not in mine (neuroscience). What, I'm going to use Java v.<whatever> on Ubuntu v.<whatever> because a funding agency tells me I have to?

I agree with much of your post, but not this bit.

edit: I think much of the professional insecurity of my field -- which feeds into the stuff you're complaining about -- is driven by the fact that way more grad student positions are funded than there are academic jobs. I wrote about this in more detail here: http://news.ycombinator.com/item?id=470181

I think lutthorn's point above is on the money. Publish the algorithms, data, assumptions, and any other implementation-agnostic information required to reimplement, but don't bother with the actual code.

Successfully reimplementing the experiment on disparate platforms would liekly serve to support the findings even more. It might be more work up front for researchers to have to do the complete implementation in their preferred platform and then work out the kinks, but it might improve the actual science being done.

> I think lutthorn's point above is on the money. Publish the algorithms, data, assumptions, and any other implementation-agnostic information required to reimplement, but don't bother with the actual code.

If you don't publish the code and I come up with a different result while supposedly using the same algorithm, data, and assumptions, how do we know where a discrepancy between my results and yours comes from?

Note that if you're taking public money, the code isn't yours. It's the publics. Don't like that? Don't take public money. (If you work for Google, Facebook, etc, they own what you do on their dime - same deal here.)

You do realize that the way the incentives are laid out, it's exactly the people who do what you ask who will not get permanent jobs, so you'll still be left with those who don't. You can't accomplish what you want by bitching. As someone said in the discussion about Google hiring yesterday, "incentives matter". I agree with you that more emphasis needs to be put on careful work instead of cranking out flashy publications, but until people are rewarded for that, things are not going to change much.

Very well said - I find the current state of publishing in academia appalling. Don't forget that most research is behind a pay wall just for the damn PDF! I think you nailed it with "insecure". The little PhDs need to know their work isn't designed to just get them tenure...

Like we choose to have our PDF's under pay walls. I don't even have printed copies of my paper, because the publisher does not want to make the expenditure. And if I were to lose my password, I would have no access to my own paper for download (and I don't have access to the rest of papers in the same issue, of course).

Couldn't you have emailed it to yourself or copied it to a USB drive when you originally wrote it?

I hope you are joking. What I mean is that access to the journal pdf is blocked. I have of course the sources, several copies and pdf's hanging around, as does my co-author.

So you're saying that you can't release due to some licensing / copyright issue, or that it's just something that takes extra time? I definitely understand the former - not that I like it, but the latter is inexcusable.

It depends heavily on the journal. Most journals have a "final draft policy": What they print is only theirs to publish. But you can self-post whatever previous versions you have. In my case, I think there are one or two minor spelling mistakes in the versions I have posted in ArXiV and my homepage. It does not take extra time (at least not a lot) to self-post it or publish on ArXiV (just a little hassle with image conversion problems, YMMV)

Would you say that you're the exception in posting them for the public? If so, would it be worth while for someone to try to get at these non-final but still perfectly useful papers?

I really can't tell. As far as I know, all people in my department publish freely his documents: either in ArXiV or in the department page for submitted papers. Also, ArXiV has a huge numnber of articles in Mathematics, the growing trend is to submit it there. I guess that most mathematicians (or at least, young ones) at least provide some draft version of their published manuscripts online, freely available.

Yea, ArXiV is a great resource, and I actually hadn't seen that it's grown this much. Your department is one of the good ones - I salute you! Here's to more doing the same.

I also hope everyone starts doing it. There is no point in making research unavailable to the public just for the sake of keeping the journal's "level". The future is open content, but most publishers are still blind to it

This is an incredibly ignorant and insulting comment. Nobody goes into science to get rich. And job security at 40 is hardly the primary goal either.

The majority of scientists go into science to add to the common pool of knowledge. For many, seeing their work being used in a positive way is extremely gratifying.

The pay walls are not the choice of most scientists. There is a push towards open-access, but it's not easy. To publish a single open-access article a scientist must pay thousands of dollars. For example, a single article in BMC bioinformatics costs US$1805. This must come out of already hard fought for and limited grant money, money that must also pay the salaries of many younger scientists still in training.

Good intensions doesn't counteract the outcome of bad incentives. Obviously tenure isn't about being rich - but it is about being independent in your work. The system produces behaviors in its "publish or perish" style that are centered on building your metrics for getting tenure / status recognition.

How many conference intros have we all heard that say, "X person has been published X times and in Y journals". Are you saying you've never heard advisors talk about splitting articles up and the like? How about the decline of repeating published experiments?

These aren't isolated incidents at all. Status seeking is in all of us, and I don't consider it a bad thing. The metrics just need to be aligned with the goals of the endeavor - furthering knowledge.

Currently, we have a system where most grad students are chattel slaves seeking PhDs for a tenure track positions that 95% won't be able to get. So competitive is the tenure system that people are doing the above things to have a chance.

It's like the idealist-activist who becomes a pragmatic-politician and learns along the way the cruel facts of life and the system. If that's an ignorant or insulting point of view, then welcome to humanity... ;-)