I think I might be quite warm naturally but those thinner caps that fit below the helmet work just fine for me when I bike in cold Sweden in the winter. -10 °C and below.

The helmet itself gives some wind protection which helps as well I think.