I think that's because 4C is the point where water is the densest, whereas 20C is a comfortable room temperature.

Interestingly enough it started out as being defined as 1 dm³ in the 18th century, then in the early 20th century was defined through water at highest density, then in the 60s back to 1 dm³.