You are oversimplifying.

While these numbers are indicators, they are non-representative samples. Different communities spread the disease to different demographics.

A CFR of 0.4% is still huge, especially if it is very non-uniform and would mean 1.2mio deaths in the US if it saturates.

And if that isnt enough for you, we still don't have reliable information about long term effects of an infection. It could end up being persistent like herpes or HIV, make people sterile or reduce lung capacity and or increase chance of cancer.

With these, possibly small, risks, it is ethically incredibly irresponsible to allow the disease spread through the whole population. You cannot gamble with a conceivable downside like that.