I've found synthetic workout clothes last pretty much forever. I have a pair of 15 year old basketball shorts that look about the same as they did brand new whereas the cotton stuff I've bought all would have holes and have faded in color by now.
These are shirts I've worn for a week straight trekking through Central American rainforests.
However the cotton fabric used in most off the shelf clothing is also not what it used to be. My dad, who doesn't like to throw things away, still has his old clothes from when he was in his 30s (he is in his late 70s now). They've retained their shape far better than my 3 year old clothes.
I've had phenomenally gross post-ride jerseys (literally crusted with sweat), but doing this they don't get stinky. I agree that the key is to not let them stay wet and let bacteria grow in it. Get them dry fast, all is good.