Papers with Code
paperswithcode.com
paperswithcode.com
I am impressed by its content. I use both for research (to track progress) and teaching. Previous references:
- Measuring the Progress of AI Research by Electronic Frontier Foundation, https://www.eff.org/ai/metrics
- Natural Language Processing Progress https://nlpprogress.com/
are great, but nowhere near Papers with Code, when it comes to the completeness, and UI.
I greatly enjoy publications that provide their implementation (or a simplified) version. Playing around with the problems yourself, can give a much greater insight and understanding in addition to the fundamental, written work.
However, I did encounter situations where providing the direct implementation was seen as a bad thing. It was thought of as `giving away your advantage' and squeezing out multiple papers before even thinking about publishing the code had their preference.
It is great to see more and more research publications go together with their implementations.
I remember going through a digital image processing course in uni where the final project was to implement a paper and check the results, and I remember that our results, when coded were different from the paper’s authors (although I can’t remember if it was because we didn’t code it like they did or if their results were not to be trusted).
It’s just so frustrating and borderline disingenuous to publish results, mention bits of code, but not include the whole code.
Decades of "printing out the code listings would cost too much paper" made many researchers get comfortable with the idea that nobody would ever see the code behind their work. That means:
- your code can be shit and potentially buggy and nobody will point and laugh at you
- replication is harder, so you can bluff some of the details
- if key findings are actually bugs, nobody will find out
These are all terrible reasons, but they're comfortable reasons as well. Open access was the first barrier, code included is simply the next.
If code included would've been commonplace, fiascos such as Imperial College of London's possibly-totally-bogus covid simulation [0][1] would've never made it to the cabinet.
[0] https://www.telegraph.co.uk/technology/2020/05/16/coding-led... [1] https://github.com/mrc-ide/covid-sim
One thing that stands out on it is that the top article is OpenAI's gpt-3 paper. While I've only read popular summaries here, one point that's highlighted is that the model involves 175 billion parameters. Which seems like an indication that while you can download the code and the paper, you're going to be able to run it - at not until/unless GPUs get much bigger (or you have Open AI, Google or similar level resources).
I wonder if you could have a site which not allows the uploading of code but actually involves the code executing on the site.
Edit: they do have heavy limits for the version they host, but the platform is open-source, so institutions can set up mybinder instances for their own users.