Yes, ventilation causes heat losses, but it is necessary.

There are ways around it though. The simplest is to not make sure ventilation goes where it needs to go. Modern buildings use mechanical ventilation to make sure every living space gets properly ventilated so one room doesn't get too much and another too little. Even better, some building use heat exchangers to heat/cool the incoming air with outgoing air, minimizing losses. Other techniques involve passing the fresh air underground, which, in a temperate climate gets you some free heating in the winter and free cooling in the summer.

Obviously, to limit heat losses, you want to reduce conduction and radiation too, which can be done without sacrificing ventilation.

You can use a heat exchanger to get "fresher" air while keeping the heat/cool inside. Although many places don't have this in place. It is probably mostly due to lack of awareness or concern than any technical reason.

In a heating/cooling system that has been specifically designed to improve ventilation, one can pull fun tricks like using the outgoing (stale) air to help heat/cool the incoming (fresh) air. Also in some places houses are built with enough thermal mass that the air within the building doesn't contain the majority of the heat therein.

In general there is likely some level of ventillation that will be worth taking on slightly increased heating/cooling costs.

Sure, but it's not nearly as costly as you'd think. We pay for all kinds of things. Including, compared to the past, much warmer air in the winter and much cooler air in the summer.

Growing up we used to put on sweaters, wear shorts, use fans, have the windows open in a car.

Changing our heating/cooling preferences to get rid of all that costs money. People don't mind.

But somehow, spending a small bit to breathe well and avoid indoor pollution/viruses is beyond the pale.