Comment by tmarthal | Hacker News Reader

Comment by tmarthal | Hacker News Reader

Parent Full thread

I used to call the scientific software that I was writing, "paper-ware".

You aren't building a system for other users, you aren't really doing anything other than one-off analysis to create charts, which will be explained in a paper.

Things have changed somewhat since the early 2000's, but the concept remains the same. Nowadays, for interesting or controversial results other scientists want to be able to verify your results. However, that is usually more related to your data and how you processed it, rather than your software algorithms (which should be explained in the paper, and can be recreated from that).

So do these systems need to have reams of documentation? Probably not. However, if you leave the system for two years and come back to work on it, or figure out how it used to work, then you best have enough commenting with a thorough readme about some of the decisions you made and why. It's more analogous to scripting rather than software engineering.

On the other hand, if you put good software out that people use, it counts as a citation. The most cited prof in the dept I graduated from maintained a widely used program for astronomical simulations.

Seriously? That's intriguing... If you wrote a very popular scientific library, how much would that impact your career compared to, for example, a Nature publication?

(I ask this because, for me, I think the former is much more likely than the latter.)

They serve somewhat different purposes, but an influential library will have far far more citations. At least in bioinformatics, that's the only way to get a large number of citations, because at their core bioinformaticians are tool builders! For example, samtools has ~1400 citations [1]. The original BLAST paper is around ~44k citations [2].

However, there are a ton of bioinformatics libraries released every year, and almost none of them gain any traction. Nature publications are far more frequent than important new libraries, and you need more political clout to get the library popular than you need to get a Nature paper.

Really, you need to gauge usefulness and interest of your library before you devote a ton of time to it. It's a lot like a startup's product.

[1] http://scholar.google.com/scholar?q=samtools

[2] http://scholar.google.com/scholar?q=BLAST

Thank you. This is the point I tried to make when I wrote the original post. What's the point of spending extra time creating documentation and examples if, on average, no one is going to use or try your software. Perhaps it's just best to create a small publication for the software then push it out the door.

I think this is more a comment on how the system is broken. Researchers should be notified of new libraries in their area. At the very least, they should be able to consult a single site that everyone uploads their code to (think Github for science with more emphasis on exploration). Academic journals are not the only channels carrying useful information.

It's not the discoverability of the libraries that's the problem, it's that the utility of these libraries is generally not that great for anyone except the authors. One common type of library handles data transformation, normalization, and maybe even workflows. These abound. But they are rarely useful in other people's hands, because to extend them and actually get any work done, you need to spend as much time learning them as it would take to write it from scratch. And the advantage of writing it from scratch is that you know it intimately, and all of its assumptions and flaws, which you don't know about somebody else's code, even if it's extremely well documented. Take something like Taverna [1], which is probably very useful to some people, and had been recommended enthusiastically to me by many people, but after spending three hours reading documents and searching the web, I could not get it to do what I needed to do, so I wrote a simple one-off bash script that interfaced with our cluster system. Alternatively I could try to hack in loops, but that's going to take me 10x as long, will require me to interact with many other people who obviously don't understand my problem since they did not consider it a fundamental need, and may not even be accepted back into the mainline, at which point I'm off on my own fork and lose the benefit of using a common code base. Waiting 1-10 hours to hear back from the dev mailing list is unacceptable when you're trying to get work done.

Is it more important to get the result, or to use other people's code? Reinventing the wheel is a minor sin compared to not getting results.

[1] http://www.taverna.org.uk

I just think that very much depends on the field and the problem domain. Taverna seems like it's more targeted towards academics that don't know how to code, and that most people that use it are comfortable staying within its limits. I mean, you definitely are going to have a level of project specificity that is much higher than say, that found in the web development world. In science, many people are searching for the existence of new problems, not just the answers. Why build a gem for email integration if the next best method of communication will likely come out next week? The problem with this thinking is that it perpetuates itself. I don't write the library that only you would find useful because I don't think it's worth my time. In return, I never receive anything useful because everyone else has adopted that same mindset. As some others pointed out, I think the problem rests in the lack of best practices and poor comp sci education among researchers. Teach proper library construction and test-driven philosophy, and I think you'll see a lot more people become comfortable writing and publishing libraries. Cobble together some basic documentation, keep an eye on its use, and contribute more accordingly. You're never going to escape writing custom scripts, but there are more well-defined problems out there that could use standard solutions.

Within machine learning, Peter Cheeseman developed Autoclass, a very popular clustering program which is probably his most highly cited work. Kevin Murphy used to be very well known, when he was a postdoc, for his Bayes net toolbox for Matlab. The author of svmlight is another example; the package was developed during his phd. I can't recall for sure, but I think FFTW started out as a student's project.

The key is to find a modular component that is difficult or novel, and at the same time broadly applicable and hopefully extensible.

Reply on news.ycombinator.com