In my experience, highest quality and best fitting button down shirt is a comfort nightmare compared to a good fitting t-shirt. Same for dress shoes vs casual wear shoes.
I recently bought two linen shirts for the first time and they’re heaven in both looks and feel.