Verifiable science on modified PCR machine
github.com
github.com
I have a few questions if the authors or anyone knowledgeable is around.
From the listed features:
Cryptographic signing of all test results
Tamper-evident resin seals on all connections and access points
Any attempts to open or modify the machine result in visible damage to security seals
> The system is designed for use in supervised laboratory environments where sample chain of custody is maintained. While the machine can't prevent sample swapping before testing, it ensures that once a sample is tested, the results cannot be manipulated.
Two questions here:
- Would an approach where you have to commit to a sample label before testing help? Before running the machine you say "ok this is sample of experiment X on patient Y", this gets written to a third party transparency log, and only then the machine will produce a result and sign it together with a reference to the log. Later you can't hide such results, or run the samples again.
- Maybe the "supervised laboratory" answers my first question of inspecting the machines for tampering, if we assume the laboratory itself is to be trusted, and that only researchers might falsify results. Is this reasonable assumption? Wouldn't the laboratory or the institution be also incentivized to at least overlook cheating?
This looks like a very interesting project in exploring how much work would really go into locking down a specific instrument. Spoiler — This is a lot of work! But, now knowing how much effort there is here, I don’t see how practical it would be.
I’m not convinced that locking down each specific piece of equipment is really the answer to the issue (which I agree is a problem). There are too many different types of equipment to really make a dent in locking down a lab’s workflow. What might work though is a resource for signing digital assets/output files to prove the raw data is unaltered. You couldn’t stop someone from uploading an altered digital file, but you’d have a digital pop trail for when a file was created. That coupled with a lab notebook, and you’d have a pretty good audit trail.
I’ve done a bit of work in this space a long time ago. The hardest part was always getting people to actually use the system. Now that journals are starting to require copies of all raw results used in figures, this type of system would make more sense and could make a real impact.
it's interesting that journals are starting to require raw data... that shift could definitely drive adoption of these kinds of systems. btw, i've seen some projects focusing on verifiable data provenance, might be relevant here.
It would be easier to take this seriously if the work was accompanied by any kind of documented threat model.
I also don't believe this is even ostensibly addressing an important problem. There's a cognitive availability bias problem in discussions of research misconduct. Incidents are newsworthy, but more people in these discussions don't have a sense the scale of the field --- which is enormous. People are talking about a numerator without knowing the denominator. A simpler way to address this problem: ignore it, unless you work in the field.
And has a massive impact on capital flows.
> A simpler way to address this problem: ignore it, unless you work in the field.
Then there's no reason to continue funding it or attempting to reason about it's output. Unless the field has a way to fund itself?
While this is beside your point, outputs from qPCR machines are often reasonably interpretable. I've seen one model that's XML, and one that's JSON, both as files in an overall data file that's just a ZIP, and I have seen intermediate files that are just plain text. In both cases, these are processed on the machine from raw images, which are a standard TIFFs, though they are usually not stored.
I am similarly left confused about the motivation of this project, however. If I recall correctly, commercial offerings of cryptographic signatures for qPCR machines already exist, running on the machines. Also, while they immediately dismiss the idea of modifying the machine, or even using different software, those are both important considerations. Many modern qPCR machines have computers running full operating systems in them (the QuantStudio machines run Android, for example, and have their core control running internally in Python); these can extremely insecure. All the work they have done would do nothing to protect against a modified machine, and contrary to what they suggest, I have found that modifying machines, and writing interface software as an alternative to the manufacturer's, can be quite easy. qPCR machines are not actually very complicated instruments, especially from a firmware perspective. In putting tamper-evident seals on the machines they perhaps don't realize how open lab equipment can be: I've seen almost intentional unauthenticated privileged remote code execution, for example.
And, as you note: it's very unclear what type of behavior this is even trying to defend against. Research malpractice is both rare, and can take place at very many points in the process. This seems like it is defending against a particularly unlikely point. I think fraud would more likely take place either in setup, or in data processing.
And finally: this locks down machines to be only usable in very specific ways. That is reasonable in diagnostic and clinical settings. But research is often not about doing the exact same thing that has already been done. It often involves new uses for equipment, and new experimental methods. Most of my use of qPCR machines, for examplee, would be impossible in their setup, as I'm using them in ways the manufacturer's software doesn't support, and the data coming out has nothing to do with qPCR, and so can't be processed in a normal way. At some level, research equipment cannot be heavily constrained, as that is contrary to the very point of research.
(For context: I maintain an open source package to control QuantStudio machines, primarily intended for using them for non-qPCR purposes.)
I feel like it's worth a reminder to those reading this that replication isn't just about preventing fraud. I've personally seen far fewer examples of fraud than I have cases where difficulty reproducing results led to the discovery of considerations affecting the experiment that happened to present in the original experiment, but that the original researchers might not have thought of, or thought significant.
Betteridge says "no" - https://www.nature.com/articles/533452a for instance...
Ultimately a human still labels the samples, and it is this labelling that is the most likely point of fraud. I.e. I label this sample as X, but it is really Y.
This is how the slightly smarter people commit fraud of their Western blots etc. (The even more stupid fraudsters do crude cut-and-paste photoshop jobs).
you could also sequence the output and see if you get what it is saying you get