My home has very poor ventilation so I periodically will open the windows and blast a box fan to "flush" the dirty air out. If done quickly, there's not much thermal loss.
It really depends on the house, if you have low thermal mass home then airing it out would definitely lead to a substantial thermal loss.