Before rushing to judgement (which is easy to do given just one reporter's perspective), it's a good idea to read Scott Alexander's experience with institutional review boards:
Injecting viruses into desperate people, on the other hand, has some pretty obvious risks. One can argue whether the current amount of regulation regarding it is excessive, but its pretty clear there needs to be some regulation.
Plus, note that the problems here weren't only regarding safety issues. The study was also poorly designed from the standpoint of getting meaningful data out of it. So even if one was indifferent to the safety of the study participants, the whole exercise seems to have been close to worthless, since the data is unusable.
* Neither could chain of custody from manufacture to application.
* Subjects who experienced adverse effects were removed from the trial without further tracking.
* People with HSV-1 were administered a vaccine based on live HSV-2.
* Subjects were tracked with self-assessment questionnaires of poor design.
* Subjects socialized with investigators and with at least one cheerleading investor.
If this story was about a pharma company, we'd be calling for heads on spikes But there's a rooting interest for underdogs (especially those up against the US regulatory state).
These questions all have purposes, and are there for a reason. For example:
"That was the first ten pages or so of the Application. The rest was increasingly bizarre questions such as “Will any organs be removed from participants during this study?” (Look, I promise, I’m not a Nazi)."
There are a number of studies done by Not Nazis that still involve removing organs.
"And: “Will prisoners be used in the study?” (COME ON, I ALREADY SAID I WASN’T A NAZI)."
I'm not a Nazi, and for one of my studies, the answer was yes - prisoners get treated at the hospital I was using EHR records for, which meant they were in the data set. As prisoners have a very distorted capacity for informed consent, it's an important question to ask.
"And: “What will you do if a participant dies during this research?” (If somebody dies while I’m asking them whether they sometimes feel happy and then sad, I really can’t even promise so much as “not freaking out”, let alone any sort of dignified research procedure)."
This is a perfectly fair question to be asked during a study - especially as for many trials, determining whether or not the death was due to the study is an important question.
As far as I can tell, this essay is "In which Scott Alexander didn't do his basic homework, but still wants to be able to do a biomedical study, and is upset the IRB didn't pre-emptively read his mind and tailor his form for his particular case."
He actually made me feel sympathetic toward the IRB...which is an impressive feat in and of itself.
As the purpose of the IRB is to police studies, then it indeed has an inherent responsibility to work at actually understanding those studies - not to simply design one-size-fits-all forms and promulgate top-down rules as its primary requirements. Such behavior reeks of an entrenched institution that does little besides bureaucratically gatekeep to perpetuate its own power (and therefore is actually shirking its intended purpose!), a trend which is eating our society alive.
- Patients not knowing what they're being studied for
- Patients signing their consent forms using an alterable substance
- Study discontinuation criteria that aren't well defined
Are genuine problems with his protocol. They might be valid (though the last one really needs to be more specific), they might not be, but the process of asking questions is how the IRB comes to understand how a study is supposed to work, and what ethical issues are involved.
I've worked with IRBs in one form or another at a dozen different institutions now. Sometimes they're annoying, to be sure, but to be frank, reading his account, I doubt I would have approved his study.
And frankly, a lot of what he's put down suggests a really dangerous lack of understanding of medical ethics. He clearly didn't pay much attention during his mandatory training.
Hell, a lot of it suggests a really poor understanding of study design.
For example, the department had a huge amount of patient information stored in their regular computer system. But the IRB demands an archaic pen and paper system for storing patient records, rather than using the computer system that exists. You could argue that maybe the department computer system was bad, but in that case it should be fixed, rather than creating a parallel system.
Similarly with the issue of consent. The doctors were already giving questionnaires to the patients and already using them to determine what the treatment should be (counselling, medication, etc.) Why do the consent requirements suddenly become completely different when it's part of a study? Should the doctors be asking patients to fill out a consent form each time they give a questionnaire? Should the forms be in pen, even though mental patients can use pends to hurt themselves? I don't know, but I do know that having bizarrely different criteria for a study versus normal medical practice seems unwise.
I agree that it would be good to have better-defined criteria for stopping the study. It sounds like Scott had an informal goal of 100 data points, but didn't manage to hit this goal.
I'm sure IRBs do some good. But it's really hard to take the IRB's side in this particular case. They weren't even following their own rules (the person asking him to take ethics training hadn't taken the training herself-- which would have probably torpedoed his study even if he had done everything else right.)
The department's clinical activities and research activities are subject to different Federal regulations, and the IRB doesn't have any authority over standard clinical care.
"Similarly with the issue of consent. The doctors were already giving questionnaires to the patients and already using them to determine what the treatment should be (counselling, medication, etc.) Why do the consent requirements suddenly become completely different when it's part of a study?"
Because it's part of a study. An entirely different ethical framework applies when you are trying to treat someone vs. when you're trying to learn how to treat them. Sometimes those criteria are laxer (for example, you can use investigational protocols and things that aren't yet standard of care). Sometimes they're not.
"Should the doctors be asking patients to fill out a consent form each time they give a questionnaire? Should the forms be in pen, even though mental patients can use pends to hurt themselves?"
Again, different contexts. They happen to be in the same locations, but very different contexts. For example, your doctor can access your entire medical record vs. a hospital's EHR system for your care. That does not mean that the doctor should be able to access all your information, for all time, for any reason.
"I don't know, but I do know that having bizarrely different criteria for a study versus normal medical practice seems unwise."
Most IRB regulations exist for a reason, and no, study vs. normal medical practice are different, because while the risks may be the same, who bears the benefits are not. The entire purpose of the IRB is to ensure the risks to the study subjects are kept to a minimum because they are also unlikely to bear most of the benefits. Medical ethics works on the ethics of a single patient.
They are different by design, and through long experience.
"I agree that it would be good to have better-defined criteria for stopping the study. It sounds like Scott had an informal goal of 100 data points, but didn't manage to hit this goal."
I meant that it would be halted if the patients become "Violent". Violent is not a clear definition.
Though he also probably should have had power calculations in his protocol. IRBs really don't like studies that don't have a chance of producing information.
"I'm sure IRBs do some good. But it's really hard to take the IRB's side in this particular case. They weren't even following their own rules (the person asking him to take ethics training hadn't taken the training herself-- which would have probably torpedoed his study even if he had done everything else right.)"
The person asking the questions are not necessarily the people who make the decisions. For example, most of the questions at an IRB I worked with were asked by admins, because they've got dedicated time to do it, and can go through a checklist for things like "Is violence well defined?" or "You seem to have deviated from the normal way this goes..."
After that, it goes to a formal committee who take a lot of training, who either approve, disapprove, or ask other questions.
The concept is absurd. "Whoa guys! A resident just entered the room! Now we have to fill out all our forms in pen rather than pencil because the ethical standards just radically shifted! Now we're learning how to treat rather than treating!"
I think you are also missing one of the big points here, which is that the study protocol was exactly the same as what the doctors were doing anyway. So philosophical tangents like whether risks should be minimized more in research or treatment are irrelevant in this particular case. The risks are the same because the protocol is the same.
You can disagree that they should be entirely different ethical frameworks, but at the moment they are by Federal law.
And the risks might be the same, but the benefits may not be. And beyond that, it's the researcher's burden to show that, not the IRBs to take their word for it.
There does seem to be a few points where ethics do directly conflict with his desires (eg getting explicit consent to be "part of a study" versus priming the patient), but reasonable stakeholders should be able to mutually work out their real concerns, rather than go off the rails with arbitrary bureaucracy - say, ask the question first for the clinical purpose, and then obtain consent to incorporate that data in the study itself.
> - Patients signing their consent forms using an alterable substance
You're really referencing the signatures being in pencil?! I don't see how this could possibly be construed as a strict ethical requirement. Please do enlighten.
Furthermore, it doesn't rise to being an ethical problem unless fraud actually occurs. And I'd be more concerned about the possibility of signatures being straightforwardly forged by a researcher ("here, let me help you write your name") regardless of a pen being used.
There's a huge difference between research and routine practice, as well there should be. Note that things that are actually done for quality improvement and routine care are exempt from IRB regulations. Even at the most onerous institution I was at, that was a one page form. At the most lenient, it was a online "Choose your Own Adventure" flow chart.
He'd know that if he had paid attention at the training past "Don't be a Nazi". Because IRB training is pretty much universally run by CITI, and I've done that training more times than I care to think about.
"There does seem to be a few points where ethics do directly conflict with his desires (eg getting explicit consent to be "part of a study" versus priming the patient), but reasonable stakeholders should be able to mutually work out their real concerns, rather than go off the rails with arbitrary bureaucracy - say, ask the question first for the clinical purpose, and then obtain consent to incorporate that data in the study itself."
One of the things that he ignores is you have to justify these things, rather than just going "I want to do it". I've been part of studies that do not even ask for informed consent due to the study design, but that's a point of negotiation and discussion with the IRB. And I absolutely think their default stance should be "No".
"You're really referencing the signatures being in pencil?! I don't see how this could possibly be construed as a strict ethical requirement. Please do enlighten."
I am. Pencil is really, really easy to alter. Having a requirement like the form must be signed in ink is a reasonable compromise between something like pencil and, as you allude to, notarized and witnessed, etc.
And his excuse, that his patients can stab themselves with pens, seems untrue for a subset of pens (felt tips) and still true for pencils.
Basically, he cast himself in the best, most put upon light as possible. I am already pre-disposed to muttering about the IRB, because it does add time to things, and sometimes is silly. That I came off reading that going "Noooope!" is a really bad sign.