Solid color or benign logo, probably OK. Band, sports team, anything jokey, college -- no, they aren't taking you seriously. Star Wars related -- they are laughing at you.
Where I'm from, wearing your football's team shirt is nothing to be scoffed at. That is, except by your rivals, but they're not taking you any less seriously - just not liking your choice of team.
And yes, it's very normal to wear that to work, especially (but not exclusively) after a big match where your team won or played particularly well, or at least bravely.
Don't even get me started about the Clarks Desert Boots.