Hacker News Reader

I'm all for research papers being made more open and available, rather than hidden away through paid-for journals --- this is in my opinion one of the worst parts of how knowledge is shared. However, I'm not sure that making the papers actually able to be modified through a platform like this will make research more accessible or authentic, at least without changing virtually everything else about how science is done in academic institutions.

If the content of a paper changes over time, it becomes more difficult to reference that paper. You now have to include both the paper and something which identifies which version of the paper you read. This is possible of course, but is a fairly big inconvenience for mostly stupid reasons, such as various citation styles not allowing hyperlinks of any form. There is also the possibility of the content of a paper changing completely, though we should hope that people would use a tool like this sensibly and it wouldn't be an issue.

This proposed platform puts a burden on the authors of the paper to maintain a paper indefinitely. Of course, any paper you author you are at least partially responsible for, and any legitimate errors should concern you. However, having to keep up with many comments and/or critiques after a paper is published would be very tiring. The metrics by which academics are measured for their employment are heavily biased towards publishing a lot of papers, and having to spend a lot of time curating old work would be detrimental to anybody's career. (These metrics are also a reason that people would rather write a new paper to build on the old, rather than incorporating it into an existing work). Another issues is papers with many authors: do they all need to agree on a change to have it merged? How often do you want to try to contact 10 people in 4 different timezones to debate about minor changes to old research?

I also don't really understand how a system like this would address the problem of fraudulent papers in a better way than is done today, which (as far as I am aware) is academics contacting journals to get the papers retracted.

I don't think it's a bad idea, but it is so far away from how people publish today that many things (journals, metrics, work loads) are against you.

I can't even get coauthors to use LaTeX+git. Word+EndNote+Emails is far more popular, despite how terrible it makes collaboration.

I think the exact tools used are irrelevant to the discussion. If this style of publishing + maintaining older papers became popular, hopefully some better tools would spring up.

For what it's worth, I think tools like ShareLaTeX and the like are much more usable for tracking changes in documents and enabling collaboration, especially with coauthors with little programming experience.

LaTeX and git are great for people who have invested the time to learn it. Any platform that will be successful for research collaboration could include these, but has to make the entry costs absolutely nil.

You could use indexes and web hooks or some sort of pubsub mechanism to auto notify researchers when a specific piece of text they referenced is changed.

I don't think that the technical challenges of notifying people are relevant here. The very fact that you would have to update your own research because something you referenced changed is kind of insane to consider. You now not only have the burden of maintaining your own research, but also staying up to date with changing prior research, and all of this work is not recognised by the institution that employs you.

If you reference something that turns out to be incorrect due to poor data or whatever, and it gets corrected surely you would want to update your paper?

For sure, I would want to. But with how institutions currently asses and pay researchers, it would be detrimental to my career to take the time required to understand the change and update my own work, which might involve dredging up years-old data and tools, contacting collaborators around the world (some of whom might have moved completely out of academia and therefore have no interest in all in maintaining work), getting them to agree on a change, etc. It might not even be possible to update my work - maybe a piece of apparatus would take too long to rebuild, or maybe the change requires that I change my entire experimental approach. It's a lot of work to do research and write a paper, and that's why there is such a massive focus on doing it once and doing it well.

I think its infeasible for various reasons to have people maintain old work in the way suggested in the original comment. Who is responsible for it, especially if the authors have moved institutions (which happens very regularly, especially for young researchers), or out of research entirely? Perhaps there could just be a shorter window in which the work could be looked at and changed if required, and after that it is frozen forever. But that actually sounds a lot like (a more open version of) the current peer-review-then-publish system.

Maybe?

It sounds like you’re imaging errors propagating through citations. Paper X cites paper Y, which in turn cites paper Z, so if there is an error in Z, X and Y are—-or should be assumed to be--invalid.

It doesn’t work like that. A few citations may be “critical” but a lot of them provide context and credit. Jones et al. first identified this problem. Smith tested several obvious but ultimately unsuccessful approaches. Here, we use the same methods as Wu and Lee. Our results differ from Cantorovich because....

Errors in those papers might make sections of the citing paper superfluous or less interesting but they don’t invalidate it. For ideas and methods, it might not even matter what the ultimate result was....

Yes, and that’s why you’ve got to push for the intitutions to change.

It's not a terribly good assumption. With programming and git (or any version control system), everything is submitted to the repository, functional features, tests, data, etc. With scientific papers, the only thing submitted is the final published paper itself. There is a countless amount of data, settings, computer code, etc, that is not included with a scientific paper.

The ability to inspect, comment, and modify a scientific publication doesn't mean much if the underlying work is unavailable.

> There is a countless amount of data, settings, computer code, etc, that is not included with a scientific paper.

And that, in itself, is a problem. How can others identify flaws in the paper if they can't see the data you used?

Open Source software includes all the code you need, not just the final binary executable. Any proposed "Github for Science" needs to do the same.

And any arguments that "the data is expensive to collect and proprietary" can be countered with "code is expensive to write and proprietary, yet open source still works".

Some data cannot be shared due to patient confidentially.

In fact, almost all medical trials or epidemiological studies use data that can't be shared outside the study.

surely there are methods of concealing patient identity? We do this with lots of other confidential personal information...

But then the research isn't reproducible...

Hashing identity solves this. All of the data can be anonymized while retaining the linkage of measurements and procedures performed on the same subject.

e.g. We don't know who this guy is, but we do know that every measurement of him was linked to this same enormous random string.

Only if your data is very limited in depth.

When doing data linking I can usually identify one person in a dataset of almost 1 million from three innocent seeming pieces of data. Pseudonimisation doesn't really work.

And any arguments that "the data is expensive to collect and proprietary" can be countered with "code is expensive to write and proprietary, yet open source still works".

These two are absolutely not equivalent. I'm all for open science, but the comparison with open source code is absurd. With the former you have to invest actual money: infraestructure, materials, suppliers, etc. and of course you have to deal/coordinate those elements. OS is... just time. There is a cost, but it is just an opportunity cost.

Most large Open Source projects are maintained by paid developers, employed at considerable cost by companies that benefit from their time. This is an "actual money" investment of considerable size.

And how many people are going to look through all the raw data you used and try to reproduce your results from it? And it's not even possible from that data to verify that the experimental setup that was producing the data in the first place was valid and and well-executed.

There are famous examples of widely-used open-source software having bad bugs (for example, OpenSSL) which no-one found, because although it was so widely used, no-one was performing careful enough analysis of the codebase. And programmers are actually paid to look at code all day. No-one working in a research institution is paid or recognised for trying to verify someone else's data or check that their research makes sense - this was the job of peer review. (Even peer review is unpaid and unrecognised at many institutions.)

open source is not a sure-fire way of catching all bugs, but it's still better than closed-source. We're talking an improved situation, not a perfect situation.

That doesn't mean there isn't a git repository (or multiple) somewhere with all the code and TeX source for the paper. It's just that these aren't usually published.

Your focus on the Tex source is pretty amusing. Imagine working for a year on a project, building countless experiments that your write code for, edit, annotate in random files or on a notepad, build huge datasets, spin up and down servers or various equipment. Then, when you finally get a result, you put write a nicely formatted document in Tex or Microsoft Word or whatever. That final document represents a tiny fraction of the work and thought you put into it. The ability to mess with that final product without the underlying 99% of thinking and effort that went into it gets you nowhere.

Seriously, this whole git idea shows the level of ignorance surrounding the actual scientific process and workflow. The data files for my work (protein crystallography) are terabytes upon terabytes large. There's no reasonable way to distribute them. They go through multiple layers of data paring. Any final raw data is included in the supplemental material, but it has already been heavily curated and reduced. And also biased.

The solution to this problem is not iteration and revision.

I'm not focusing on the TeX source. I said "code (i.e. simulation and/or analysis code) and TeX source (including the original images for plots and the data those were made from)".

I once suggested the same idea for bills in Congress. They use to stuff those little addons to the bills for billions of dollars in spending and you could not determine who did it.

I have a slightly another idea. Any research activity of a scientist should be recorded by him/her in his/her own repository. Any activity, like reading a paper, or moments like "I have an idea", or "lets try a small scale pilot experiment with fellow scientists of mine as a subjects", or "pilot experiment was unsuccessful, but can be modified...".

Any publication should be forked from the main repository, selecting from any relevant data from parent, and in that fork strict procedures of scientific research are recorded, processed, and finally end up with an article.

In this case you can track how an idea was formed, how it was modified to match empirical data. You can track when methods of data processing was chosen (before gathering of data or after), and so on.

Such a repository can give enough information to decide was the research is a fraud or no.

It would also heavily detract from actually writing the paper. It's easy in hindsight to think about certain steps it took to get where you end up, but if you are at the start, there are simply too many intentional and unintentional steps to record everything. I much rather have people focus on the actual science than documenting their steps ever few minutes.

An article is a hindsight reflections on what was done. If you are doing research you need to make notes. I personally can't remember what I've read and where I got an idea. I need to write things down, to be able to refer to it later. I'm logging my thoughts on regular basis, just to be sure they are verbalized and formalized. The goal is not to document every step in a formal manner, the goal is to leave a trail that can be tracked by a curious researcher to see how article was formed.

Current situation is bad, because articles do not capture all the information that needed to validate the research findings. p<0.05 is not enough for this. Including more information into the article is one of the ways to deal with that problem. But the other is to leave footprints, to create some historical data, that could be analized. Or not analized if no one needed it. It is like lazy-evaluation: just generate some data to allow your followers to reason about what precisely you've done. If they needed it, they will research your trail. You do not need to go through a difficult task of writing a high-quality texts about every your step.

The only thing you need to change is to make such notes public. It is coupled with some potential ethical issues, like privacy of subjects for example. If I run experiment, I gathered some data, I couldn't just throw this data into public access. It is possible that there are other ethical issues, maybe something about privacy of researcher as a person. These issues can make logging research activities a pain.

It's a bit rich for this to be proposed in a software-centric forum when our industry is merrily rebasing its way to clean linear histories that record the story the author wants to tell rather than what actually happened.

That is the idea behind the lab notebook (what was done) + journal (what was thought) system that people are supposed to be following. We only know about Gregor Mendel because of his meticulous notebooks, for instance. My boss taught me the easiest way to write a paper is to start writing it before your first experiment, references and all; it's a great way to find weaknesses in your approach before you go too far in one direction or another. The problem is people get lazy documenting everything, especially when you are trying to leave the lab as fast as possible at the end of the day :)

I am not sure about this... are you talking about experimental research? I am a theoretician. I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues.

The advantage of theoretical papers is that the reasoning is a proof/argument developed in the paper itself. With theory, there's less chance for a fraudulent paper, even though the results may have gaps.

> I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues.

Thats the point. If you published your preliminary silly idea and your colleagues didn't find immediately that it is silly, then your not sillier than they are. :)

> the reasoning is a proof/argument developed in the paper itself

Even when I studied math I found sometimes that just to have proof is not enough to understand. The only way to understand sometimes is to learn how the proof was devised. What the hell author thought to get an idea of some trick in the proof? Some tricks looks as a magic, you see they are true from a logical standpoint, but they make no sense. You cannot understand how they lead to a QED. You can verify proof by logic -- not a problem, but you unable to understand. And if you are unable to understand, then at least you are unable to reproduce this proof at the exam, the only way is to learn it by heart. By I never trained my memory to learn by heart, it is really hard for me and annoying, and moreover I do not believe in learning by heart: to master a subject you need to understand it, not to learn it by heart.

My math experience taught me that if I cannot follow the thoughts of other mathematician by just reading his proof, it means that he is smarter than me, so it doesn't matter then how stupid he behaved while searching for his proof, he outsmarted me already.

But yes, I was talking about experimental research.

> I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues.

I think this stigma against making mistakes is a big problem, as is the ego-investment. We're all stumbling around in the dark, and we shouldn't be afraid to admit that.

It’s common practice for scientists to keep logbooks for this exact reason. This is already done.

As I understand that logbooks are not available to other scientists, and if group of highly qualified scientists are making a meta-research, combining results of a dozen of articles (and rejecting two more dozens from their study), they cannot use logbooks of articles to reason about articles.

It's true that more open-access logbooks might improve things. But even if you had the logbook, you still wouldn't have the custom equipment, experimental apparatus, experimental subjects, and oversight throughout the experiment which you would need to verify that the logbook itself has data worth a damn.

At some point, you have to trust other professionals working in your field to do their work well. Or rather, decide who to trust.

Maybe you are right, but I have a chat with a person who did meta-research and as I understand they dig into articles, looking for methodology, and if they found methodology good enough they would incorporate that article into their meta-research.

It impies trust in good intentions of others. But not the blind trust that anything was done by a scientist was done good.

If logbook can be used to verify that data processing methods was picked before the first experimental subject came to a lab, then it is already a good sign. It could be much harder to do some p-hacking like [this](https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...).

I rather see reproduction platform where someone can use a paper and create a new paper or fork some parts of it with credit. The original paper should be immutable.

Subversion would be the obvious choice, not git. The whole point of a DVCS is that it avoids having a canonical repo, whereas in science we very much want to have a canonical version of the truth (otherwise there would be no such thing as fraud, if everyone is allowed their own version of the truth), and Subversion defaults to having a canonical version.

I agree. As I said probably the tools and technology should be tailor for the specific use case of scientific papers. It would be nice however that these papers evolve online, instead of the written in stone papers that are publish today in scientific journals. A small note instructing the reader that a certain theory has been disproved or changed in the original paper would already help.

Peer review is not about reaching consensus tho, is about verifying methodologies and reproducibility, a system like that could easily become a tyranny of the mediocre, see for example Wikipedia.

There are certainly initiatives going this way. E.g. LiveCOMS in my own field. The crux of the matter is whether these will gain traction or not.

https://livecomsjournal.github.io/about/

I would drop the concept of "paper", and use a unique, gigantic graph of hypothesis and experiments that keeps growing as research goes on.

Everyone would contritube to the same graph without duplicate data, or unaccepted piece of proof/knowledge

https://veritas.icu/blackpaper

It is hard to make happen in competitive fields because good data is an expensive advantage over other labs.

It is a great idea. However, we do not have a distributed free internet throughout the world.