When I finally starting doing some swimming in a pool I was amazed at how much easier it was to swim. Especially maintaining momentum. I also swim quite frequently without a board in open water when the waves are small, but there is still light chop and although it's surprising how slow it is I've never felt it was particularly difficult. Interesting how things work much differently transitioning the other way.
Also when you talk about wetsuit, it sounds like you haven't used a good/thick wetsuit. Or maybe this was a while ago. Wetsuit technology has been pushed really far in the last 5 years. Dry suits have always been good thanks to the military, but sports wetsuits have really progressed a lot with the explosion of the surf industry. I stay in the water easily twice as long now than I did with my first suit I bought five years ago.