The day when code used to produce a paper must also be published can not come soon enough.
And if that means scientists need to learn about version control, well... they should if they're writing code.
But commit ids work just as well.
For what it's worth, using results from outdated source code is extremely suspicious. This is a frequent problem in software development where we have tests or benchmarks based on stale code, and it's almost always incorrect. I would not trust your results if they are not created with the most up to date version of your software at all.
- Use known, plaintext formats like LaTeX, Markdown, CSV, JSON, etc. rather than undocumented binary formats like those of Word, Excel, etc.
- Keep sources in git (just a master branch will do)
- Write all of the rendering steps into a shell script of Makefile, so it's just one command with no options
- I go even further and use Nix, with all dependencies pinned (this is like an extreme form of Make)
- Code for generating diagrams, graphs, tables, etc. is kept in git alongside the LaTex/whatever
- Generated diagrams/graphs/tables are not included in git; they're generated during rendering, as part of the shell-script/Makefile/Nix-file; the latter only re-generate things if their dependencies have changed
- All code is liberally sprinkled with assertions, causing a hard crash if anything looks wrong
- If journals/collaborators/etc. want things a certain way (e.g. a zip file containing plain LaTeX, with all diagrams as separate PNGs, or whatever) then the "rendering" should take care of generating that (and make assertions about the result, e.g. that it renders to PDF without error, contains the number of pages we're expecting, that the images have the expected dimensions, etc.)
- I push changes from my working copies into a 'repos' directory, which in turn pushes to my Web server and to github (for backups and redundancy)
- Pushing changes also triggers a build on the continuous integration server (Laminar) running on my laptop. This makes a fresh copy of the repo and tries to render the document (this prevents depending on uncommitted files, the absolute directory path, etc.)
Referencing a particular git commit should be enough to recreate the document (this can also be embedded in the resulting document somewhere, for easy reference). Some care needs to be taken to avoid implicit dependencies, etc. but Nix makes this much easier. Results should also be deterministic; if we need pseudorandom numbers then a fixed seed can be used, or (to prove there's nothing up our sleeves) we can use SHA256 on something that changes on each commit (e.g. the LaTeX source).
For computationally-expensive operations (with relatively small outputs) I'll split this across a few git repos:
1) The code for setting up and performing the experiments/generating the data goes in one repo. This is just like any other software project.
2) The results of each experiment/run are kept in a separate git repo. This may be a bad idea for large, binary files; but I've found it works fine for compressed JSON weighing many MBs. Results are always appended to this repo as new files; existing files are never altered, so we don't need to worry about binary diffs. There should be metadata alongside/inside each file which gives the git commit of the experiment repo (i.e. step 1) that was used, alongside other relevant information like machine specs (if it depends on performance), etc. This could be as simple as a file naming scheme. The exact details for this should be written down in this repo, e.g. in a README and/or a simple script to grab the relevant experiment repo, run it, and store the results+metadata in the relevant place. Results should be as "raw" as possible, so that they don't depend on e.g. post-processing details, or choice of analysis, etc.
3) I tend to put the writeup in a separate git repo from the results, so that those results can be referenced by commit + filename, without a load of unrelated churn from the writeup. This repo will follow the same advice as above, e.g. code for turning the "raw" results into graphs, tables, etc. will be kept here and run as part of the rendering process. Fetching the particular commit from the results repo should also be one of the rendering steps (Nix makes this easy, or you could use a git submodule, etc.)
I don't know what the best advice is w.r.t. large datasets (GBs or TBs), but I've found the above to be robust for about 5 years so far.
Basically, the reward system is designed to be easy to measure and administer, but is not actually useful in any way to the advancement of science.
This is obviously a shame, I was working on segmentation of open wounds and most papers include a "we are currently in talks with the hospital to make the data available". If you contact the authors directly they will tell you that their committee blocked it because the information is too sensitive.
Something like: "To make it easier to verify the code behind this paper, we've used <accepted standard project/practice> to generate a synthetic dataset with the same fields as the original and included it with the source code. The <data-owning institution> isn't comfortable with publishing the full dataset, but they did agree to provide the same data to groups working on verification studies as long as they're willing to sign a data privacy agreement. Send a query to <blahblahblah> ..."
This would be administrative overhead, it will be shut down 9 times out of 10. I understand why this might seem easy but it really is not, you can have multiple hospital that each have their committee that agreed to give the researcher their data. They don't have a central authority that you can appeal to, much less someone that can green light your specific access.
As for the synthetic datasets that's basically just having tests and was advocated for elsewhere in this thread.
I didn't say it was easy--I just said it struck a balance relative to trying to openly publish the whole dataset. Yes. Obviously comes with administrative overhead. So did dealing with the initial researcher. If the institution can manage the one, it can manage the other.
> As for the synthetic datasets that's basically just having tests
An appropriate synthetic dataset would inevitably be part of a great test suite, but it's also pretty simple to write narrow unit-tests that embed rather than stretch the same assumptions and biases that are also in the code (i.e., simple enough that even people who feed themselves with code do it).
An independent project/practice for synthesizing sample datasets from the real dataset lowers the bar and clarifies the best-practice for releasing a dataset that a verifier could actually use to spot simple bugs, edge-cases, and algorithm issues. Ideally, yes, this practice nudges the researchers to bother running their program over generated sample datasets as well, and to pay attention to whether the results make sense.