The Mythbusters checked the increased viscosity myth (opposite presumption, same speed is possible) and found it plausible.
It turns a research group at UMN actually was crazy enough to do the test - it looks like the effects balance for a human swimmer.
http://it.umn.edu/news/inventing/2004_Winter/goingforthegoo....
It's easier to maintain momentum and glide in cold water, whereas swimming in warm water feels like you're swimming in molasses.
Also, as my body heats up, cold water feels more cooling, obviously.