What do you get if you publish a paper in a highly-ranked journal?
blogs.sciencemag.org
blogs.sciencemag.org
I've talked about it before on HN but I will only publish to open access and open data (most important) journals that require all data be scrubbed of PII and published alongside the paper for replication purposes. I highly favor journals that not only allow, but encourage replication (yes, many elite journals cite "novelty" as a top factor regarding publication decisions... embarrassing).
Science should be freely available, open access, open data, and replicable. Otherwise it isn't science. It's primarily garbage that exists to advance the careers and egos of a select few.
The only reason I can think of this unwillingness to publish codes is the fact that these codes written by physicists are often extremely unreadable: very long Fortran routines with awkward variable names, no comments, no test cases, no documentation… Once you get a result from your own software, you get more satisfaction if you publish a paper with the results than if you polish and publish the code.
IMHO this is bad science, but it is difficult to change this way of working: cosmology is today done by large collaborations, not by individuals, and if you propose to make the codebase developed by your team public, this idea is usually not welcomed by your co-authors.
[1] http://ascl.net/
"All computer codes involved in the creation or analysis of data must also be available to any reader of Science."
And
"Upon publication, Nature Journals consider it best practice to release custom computer code in a way that allows readers to repeat the published results."
Imagine that the bundle starts with a manifest file that describes the data and the code, pointing to an authoritative, versioned copy. The user can "run" the manifest, which will download and install the data and any necessary dependencies. (If you're thinking about this from a containerized point of view, the code dependencies and data may be modeled as a container or virtual machine image.)
The manifest is like a build file that executes the code passing the data as input, in a clear and reproducible way. Anyone running the manifest will get the same results as the researcher did. Because the entirety of the execution environment is versioned and captured by the manifest, including all of the system software, the results are reproducible from the code and data phase onward with zero effort.
For bonus points, the manifest would also "build" the paper (e.g. if it's TeX), and would substitute the results of the code execution directly into the paper, i.e. graphs and numbers. You could conceptualize the paper as something like a notebook, where its values can change dynamically according with the data you provide it.
That's why algorithms are written in pseudocode, and mathematics is written in, well, mathematical notation. It's expected that if you want to use it you'll re-implement it yourself in a software setup convenient to you.
So you can't publish runnable code unless it's severely limited - being in some standard language that is unchanging. I'd much prefer someone tell me in words what statistical method they used, so I can type it into Python myself than be forced to spin up some standardised but old and crufty fortran 77 or something that they didn't enjoy writing and I didn't enjoy running (I am aware that Python is often just calling the old fortran libs though!). Giving me the Python in the first place also isn't feasible - some scientific analysis code I wrote 6 months ago already doesn't run because of API changes in matplotlib.
A few years ago I saw Python 1.5 code implementing an algorithm I wanted to use. I couldn't run it - instead I read it, and I read the Python 1.5 docs about the bits I was unfamiliar with. If it were in pseudocode it would have been more self-contained.
Code and data formats, other than very crufty ones, are living. They're not suitable as the final form to publish in. If your project and field is ongoing, then by all means try to develop standards and use the same software and formats as each other to help sharing data. But the actual publications need to still be readable in a decade (or much more) time, so publishing runnable code seems like it conflicts too much with that.
code => a Docker/CFI image.
dependencies => captured in the image.
format => the code understands the format.
However, the costs are fairly comparable to what was once the cost of keeping paper copies for everything, so I think it is a cost that can be absorbed by academia.
Docker/CFI is very popular, hard to believe it will disappear short of a complete breakdown of the digital era.
My point is that we've done a pretty good job at archiving and deciphering ancient text so far. There's no reason to think that we won't be able to emulate current CPU architectures (and run virtual machines on them) for a long time. Pseudocode works for algorithms, and it worked for your case, but it doesn't work for the 99% of scientific software with more than a thousand lines. Executable papers are absolutely possible and are currently being created with virtualization/containers. It's just not easy yet.
Best practice here is to self-host + host on a third-party site and run a torrent as well for the archived files. I've done it in published papers and one of the OA journals I publish with strongly recommends this exact path.
I'd represent the code as a virtual machine image. Let the researcher work however they want. When they're done, they take a virtual machine snapshot of their execution environment. We provide some tooling so that running the "manifest" in this execution environment (re-)produces all of their computational results. Thus, think of research as a "build" process that runs analysis and produces output. Ideally this build would also compile any necessary source code before running the compiled applications in the context of research data.
Far from being "limited", researchers can run any code that can run on a cloud hosting provider today. To ensure portability, the VM image will include a versioned copy of (or reference to) the kernel (e.g. Linux) and all of the userland libraries that are used by the software (e.g. Python). Think of it like a self-describing Linux container, like a Kubernetes Pod [1] or Docker Container. The image fully describes the software running on the system; it's a precise copy. With this machine image, we can run code in the same execution environment that the researcher used.
Have you ever run a very old NES or SNES game on a Nintendo DS, or on your PC in an emulator? It's the same concept.
The researcher's data is stored directly within the virtual machine image, or is represented as a virtual data drive that's mounted in the virtual machine. When this approach becomes influential and widely adopted, researchers will represent their code as containers throughout the development process. The researcher won't just run Jupyter (or Matlab or whatnot) on a random machine, they'll e.g. run Jupyter in a carefully packaged minimal container that was designed for reproducibility.
> Code bit rots.
"Bit rot" is a phenomenon that references the difficulty of maintaining code as environment and system change around it. It doesn't directly apply to our scenario. A virtual image containing a Python 1.0 program running on version 1.0 of Linux will continue to work indefinitely, just like an old SNES game. New software releases don't affect our ability to run old software -- they just make it harder to mix the two together. Furthermore, we can even run the program while passing different (modern) data as input! We've already made huge progress over where we are today.
Sure, if we want to adapt that code to a newer version of Python and Linux, then we have work to do, but that's a different problem than reproducibility. There's no free lunch; nothing can make that problem go away. But if we do want to adapt the researcher's algorithm to another language or platform, then we have a huge advantage: we can run the researcher's actual code with their actual data and get the very same result! That's huge! That will make it far easier to confidently adapt their algorithm to a new environment.
E.g. "Large cash bounties paid for highly-ranked journal articles in China and other Asian and middle eastern countries".
Most comments are answering the question by stating the traditional reward. The article is about how very different rewards are appearing.
"The next question is whether there’s anything wrong with this idea, and if so, what. It makes me uncomfortable, but that’s not the appropriate measure. As long as the journals themselves keep up their editorial standards, the main effect would seem to be that their editorial staffs must get an awful lot of China-derived manuscripts whose authors are hoping to get lucky."
Well, I'd offer that this analysis ignores how science has historically worked.
Essentially, science has involved something like a "club" of individuals who can be trusted to make a sincere effort to find the true. Certainly, the use of exact instruments and the theoretical reproducibility of experiments were important but also dealing with individuals are reliable, systematic and trustworthy was an important thing.
Which is to say that science journals aren't designed to filter out articles which are wholly fraudulent, a tissue of lies cleverly created to mimic an actual scientific breakthrough. With a clever enough person, naturally, there is no way to see fraud from the article, one would have to lab at the laboratory, attempt to reproduce the experiment and so-forth (and reproducibility was the key historically, most experiments weren't actually reproduced and no journal is going to maintain a lab to reproduce submitted articles - except perhaps in CS where a program might submittable).
And there you have it. The lab coated scientist is a movie character but be scientist, as least once, was more than to engage in some activities. It was to be part of the "broad march of progress" where the lesser-scientist still helped move things forward by small, honest iterations of basic research (and this "club" arguably was disproportionately and unfairly white, male and first-world but every process has its weaknesses).
A move which makes the pursuit of science a purely adversarial affair, by consigning poor performs to absolute poverty and giving fairly vast rewards to good performs is going to break the club everywhere.
And the implications of the end of science as an idealist pursuit vary from field to field. Fraud in CS or math might be impossible or might be rendered impossible with suitable measures. But in social sciences or other fields, things could get nasty indeed.
Honestly, I suspect people fudge thing in math more than you might think.
That said, a tool like coq will deal with any inaccuracies as things good forward. Some portion of math has supposedly been verified with it.
But, of course, I haven't run the program, just read about it. So still - no dog in that particular race.
I don't believe there are really hero superstars in science. Sure, there's a gradient of ability, but just because you happened to be the first one to discover something important doesn't mean you'll always be the best at future discoveries. There must be a lot that depends on the environment they work in.
We've been trying hard to cure cancer for decades, with significant progress but we haven't cut the death rate even in half yet. We can't get people to Mars. Phones still need batteries charged every day or two. Most cars still burn fossil fuels. We don't have clean, safe, nuclear power. We don't understand nearly everything about how the brain works, and millions of people suffer from poorly controlled mental illnesses. We don't have household robots to cook and clean. It takes a whole day to get to the other side of the world.
These are all bottlenecked by science. 2x more scientists would result in substantial improvements in daily life within a decade. 10x would be better.
- It incentivizes fraud.
If the only way to avoid a McDonald-level job is a ground-breaking paper, you'll be willing to fake that ground-breaking paper, especially since if you're caught, you can still get the McDonald-level job sooner or later.
I am referring to a lot of current conditions, where adjunct professorships are paid at close to bare survival wages and "superstars" get substantial compensation.
you have my admiration twice, then.
it doesn't count for much from me to you, i know.
but improving the human body of knowledge is, in a wide sense, its own reward.
However, people still had to like you in the interview and job talk.
I am truly surprised your faculty search wasnt more fruitful, given there is hardly much more you could have done other than maybe secure a K99 grant.
Why do you suspect your cv was not top-tier competetive? I ask because yes, I am in Academia and from my experience (including being on faculty search committees) suggests you would have at least been invited to job talks.
Grad school was successful. However, my postdoc was a different story, I got a relatively prestigious fellowship at a DOE lab but my supervisor quit before I started and I tried to do my own stuff but really didn't get much support and it wasn't much fun or productive. I quit after 2 years, about 1 year ago and joined a small biotech company in the bay area. I'm pretty sure I already have too large a gap in my publications to go back, I thought about going back to academia for a bit but that would have entailed doing a 2nd postdoc but I really wasn't interested in moving some random place for 3+ years, then dealing with the whole faculty application process again and 5+ more years before eventually getting tenure.
If you don't hate writing grants maybe academia can be for you.
Good luck.
I would imagine the same applies to most jobs - rather than an on-the-spot bonus, if you published in a prestigious journal as part of your job, it would be strong grounds for some compensation at your next review.
It also means that the position of the traditional, subscription-based journals are cemented more, even though many funders are also aiming to transition to open access publishing.
So overall, I guess I'm not that enthusiastic about this.
[1] https://www.nature.com/news/why-high-profile-journals-have-m...
2) Grants!
3) becomes easier to attract good postdocs, which helps you publish good papers faster, which helps...
4) Grad students
[Source: Pillow talk]
For example, let's say you develop a novel modeling methodology for a company who hired you. You'd like to publish the methodology and give conference talks on it. Since you're hired, your work and the methodology would belong to the company as their asset.
Are there certain types of licenses applicable here? How about in the case of the article where researchers are compensated for presenting a talk on that methodology?
This is negotiable upfront for industrial research positions. You should always review your IP contract inside your job offer packet and battle HR/management hard on the rights to the IP and compensation surrounding it. Most boilerplate language assigns all IP rights to the company you work for with a maximum of $1 consideration paid for your work (to satisfy contract law if it is ever argued in a court), with acknowledgements or minor credit to you in the official documentation.
Needless to say, you shouldn't blindly agree to that unless your base salary is commensurate with such a loss of control and future rights of your work.
Is there a licensing vehicle that allows for publishing while remaining an asset of the company?
For example, Square built their Dagger library and uses the Apache license. http://square.github.io/dagger/ My impression is that Dagger is still an asset of Square, but you can use it as long as you meet the licensing requirements.
Is there an equivalent to this for a modeling approach or work done by a consultant?
I'm trying to understand what kind of legal transformation takes place one a paper on a non-patented methodology is released. I've skimmed the dynamo paper and it seems a good analogy in that it's about general database architecture concepts, rather than the specific implementation at Amazon. I might be wrong though as I haven't read it deeply.
For example, let's say you're the primary author of a paper and then later leave the company. Can there by any legal restriction preventing you or others who have never worked at the company from giving a conference presentation on that paper once published?
Just wondering...
If you're going for your Ph.D. then your goal is to further the art.
Technical development and design that results in some "method and apparatus" to do a particular thing is patentable, but the research required before that R&D that leads to understand how stuff works and what might work is not patentable. And that's not speaking about less technical fields of science where nothing is patentable, or computer science which is not patentable in much of the world (software patents aren't valid in EU and other places).
Also, patenting is expensive. Especially because you need to file a patent in multiple major markets, you need multiple different patent lawyers, if you're outside of USA then filing a patent in USA is a bit of a hassle, and just for USA+main EU countries gets quite costly quick - so that's a reason not to patent things unless you have a clear picture that some product is going to violate that patent and thus it'll become valuable.
In my experience, the lawyers just chuck 'A System for' in front of the paper's title and open the firehose at the patent office. Lots of ridiculous stuff gets through...
Possibly one depicting a kitten which says, "me-wow!"