Returns are always a pain in the butt. No matter how easy you make it sound, people are going to be anxious about it. But even assuming returns are effortless, they're a HUGE money sinkhole for online retailers (Zappos returns average 35%).
And a huge time sinkhole for me. I have a few tshirts that don't fit very well and didn't bother returning because I didn't want to take the time to go to the post office and wait for the package to come back.
Do you have a link for the return rate? Not that I doubt you, but it'd be some great data to have.