Hacker News Reader

I disagree that ending trials early is solid science, or at least not _as_ solid as running them to completion. It makes it hard to know whether results are due to a treatment effect or just random fluctuations over time. Essentially, it allows for cherry-picking of significant results in a way that bypasses family-wise error correction.

You can quantify, perhaps by simulation, the amount that it's less solid.

What's balancing against that is continuing the trial means giving patients a placebo when you have evidence of a better treatment.

It's a really complex issue both scientifically and ethically.

The solution to this is changing the stopping criterion in the trial protocol description, so that the real confidence interval can be calculated for the trial _with_ stopping (which is very different from running the trial until the end).

Good point, I should have added prospectively to that.

But even then, more participants and more time for follow-ups is generally better all other things being equal (which they never are).

Things got extremely political around these experiments.

When 2 days ago Orban announced from nothing that they will provide Remdesivir to everyone, I knew that again they did something stupid...I want the experimental drug that Trump got, not some leftover drug with lots of side effects.

Now I know that it was not Orban who was screwed, but the whole EU.

As I said, ideally RCTs would run for as long as possible. But in meatspace we cannot ignore that if a treatment shows serious promise it is unethical to deprive control group patients of the treatment. If we ran RCTs for all CoVID-19 treatments for something like 3 years before using any treatment, it seems pretty apparent we'd be looking at a significantly higher death toll.

My point about it being solid science is not that it is a good (from a scientific perspective) to end trials early, merely that the early part of the trial was still an RCT and the data is just as solid as any other RCT which ran for the same period of time. From what I've seen, this is a contentious topic in academia because the ethics of this make it harder to argue that RCTs should always be run to completion (with no exceptions).

But any good RCT will have a stopping criteria established before the trial starts to eliminate ex-post-facto cherry-picking. I'm not familiar with the exact details of the Remdesivir trial and whether they had a reasonable stopping criteria defined beforehand -- I was speaking more broadly about the topic of ending RCTs early.

There's a well known pattern that studies conducted by a drug company are more likely to report positive results than studies conducted by a neutral party.

The study you links to uses death or heart failure as an endpoint. If there had been lower mortality or ICU admission halting the trial early would make sense. With the only improvement being quicker recovery the value proposition of the treatment isn't that great. From a practical standpoint remdesivir is expensive and had supply issues so halting the trial early doesn't mean every American is going to start getting treated with it.

I don't think the science is solid when a trial can pick from 2nd-tier endpoints and stop the trial once an improvement is found. It is quite possible that if the trial had gone on longer the one positive effect would be reduced or even go away entirely or that we would see a negative effect in a more important endpoint.

In urgent circumstances where the demand for hospital services are high, a twenty percent improvement in recovery times is the same as increased capacity.

Nit: It's not quite the same. If need for hospitalization is growing quickly, cycling out people 20% quicker might not affect overload dramatically.

Steady-state, sure, you can treat 25% more.

The fact remains that it is standard procedure to halt trials early when promising results appear, and people are dying waiting for a drug. This is not something that's controversial, and this is exactly what happened with the previous Remdesivir study.

Of course it's disappointing to see it may not be as effective as we previously thought. And of course analysis may yet reveal issues with either one or the other or both studies. That's how science works - facts, not heartfelt desires.

Nobody is dying. The only endpoint that improved was recovery days.

> Nobody is dying.

Perhaps we are talking about different things? I was addressing the current COVID-19 pandemic.

> With the only improvement being quicker recovery the value proposition of the treatment isn't that great.

It's $3k a patient. The five days difference in hospital stay that was implied costs a lot more than that, even if there is no mortality/morbidity benefit, and that's ignoring that we were worried about running out of hospital capacity.

That assumes the effect in the study is real and that there is no increase in mortality or ICU admission: unfortunately stopping the trial early casts doubt on that.

I'm speaking just to the specific point I replied to. That's why I couched it with "that was implied".

Secondary endpoints, decisions about when to end trials, etc, are all fraught with peril.

It's worth noting that neither SIMPLE nor the ACTT-1 Remdesivir studies moved from a mortality to a non-mortality endpoint. SIMPLE did have an endpoint change before any data was available, but it was a relatively minor, innocuous thing.

> The primary efficacy end point was clinical status assessed on day 14 on a 7-point ordinal scale consisting of the following categories: 1, death; 2, hospitalized, receiving invasive mechanical ventilation or ECMO; 3, hospitalized, receiving noninvasive ventilation or high-flow oxygen devices; 4, hospitalized, requiring low-flow supplemental oxygen; 5, hospitalized, not requiring supplemental oxygen but receiving ongoing medical care (related or not related to Covid-19); 6, hospitalized, requiring neither supplemental oxygen nor ongoing medical care (other than that specified in the protocol for remdesivir administration); and 7, not hospitalized (see Table S1 in the Supplementary Appendix, available at NEJM.org).

> five days difference in hospital stay that was implied costs a lot more than that

In case of COVID, it's not just a cost issue, but also clearing the bed as soon as possible, to put a new patient in.

10% less hospital stay, means 10% more patients able to get help.

Read the whole comment. I said:

> ... and that's ignoring that we were worried about running out of hospital capacity.

Anyways--

> 10% less hospital stay, means 10% more patients able to get help.

This isn't true. It's only true at steady state, but if there's an exponential process you get a smaller benefit. (E.g. take the ridiculous case that 100% of patients show up on one day-- obviously a shorter stay doesn't improve the number you treat in that case).

We're talking about a real virus here, not a theoretical experiment in math. Currently, most of europe is having an exponential growth of new cases, so even if 100% of current cases show up on day one, we'll have more new cases tomorrow, and any way to speed up the process will save lives.

Yes, shortening the process saves lives. I don't disagree with that.

It's just worth noting that you get significantly less than a 1/.9 benefit in capacity when you shorten hospitalization times by 10%. It's a benefit, but it's only this ideal value when things are perfectly steady state. So sorry for the pedantry, but "10% less hospital stay, means 10% more patients able to get help."--- it's not the predicted 11% or even 10%.

The trade off between giving access to medicine that clearly helps and testing its effect carefully isn't done very well if you stop early with only a minor effect size. As at that point you're more than likely to just be giving people access to medicine that is far from clearly helping and possibly not even vaguely effective.

What is the _reason_ for ending it early? Is it the cost? I imagine it must be high enough because then they're gonna have to fight with this very criticism later on: RCT ended prematurely, therefore it is not as reliable.

A question to you and grandparent post: what are the things that distinguish a high quality RCT from a poor one? The number of participants, the design study, etc.? Any links/resources that discuss this would be appreciated.

edit: ok, I found a paper on this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:tex....

I am super curious now about any discussion that reveals why the WHO study was less-than-stellar, and why the original RCT was considered high-quality.

Its typically an ethical decision. In the event of a deadly disease ( cancer, covid, hiv etc ) its considered harmful to continue giving a less effective treatment once you have shown some superiority/inferiority.

The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early.

Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned.

There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).

The reasoning is money, the unreasoning is ethical sophistry.

if you don't think the quality of the trial is so bad as to outright impair approval (and right now the political pressure for a tool, any tool, to fight COVID is so intense that there's zero chance it wouldn't be) then why continue? It costs money to do trials and it represents lost revenue where you could be the sole seller of the only known drug to treat a deadly pandemic.

in fact there is a downside risk to continuing as well, further study might show that the early results were an outlier and the drug is not as effective as previously thought (or has some serious side effects in some situation), and then you lose all the money you've invested. But if you push it through, there is no risk of being sued since the US Government has waived any liability on the part of drug companies for these therapies.

in a game-theoretic sense there is absolutely no benefit to continuing the trials and potential downsides to doing so. Remember, for-profit companies don't care about maximizing public health, they care about maximizing revenue. Especially when you've explicitly removed all financial liability for being risky with public health.

It’s both ethical and statistical. If you observe a strong enough effect, you can reach significance with less data. When people are dying, sometimes you throw a Hail Mary.