It seriously takes me 2 seconds tops to plug or unplug. That's zero as far as getting in the way of my life. And it isn't necessary to do this every day either.
Have you actually timed it? I guarantee it takes you more than 2 seconds to grab the plug off the hook and plug it in. How much longer I can't say though, it is case by case.