Not meaning to be inflammatory but the same scenario exists in flipped terms: how many people would you let die of cancer rather than, idk, eat the downside of running riskier experiments.
There’s no privileged neutral really. Consequentialism just owns that. Now, that might be a separate question from whether society should act as if there’s a privileged neutral (ie be nonconsequentialist).