You're merely reinforcing their point. Its so out of fashion it would be considered a bold or even edgy choice just as dressing casually once would.