Swimming uses more muscles and requires more concentration. I had a large amount of form degradation when I took a years' break swimming, and it took a few weeks to get back in form. I was noticeably faster after I regained my form -- I'm not sure what the equivalent would be on a bike, since theoretically I would just need to pedal faster (requiring more raw strength, not form.)
I don't think there's much comparison to the bike example in the OP.