> But once you reach a certain age the social stigma of it will prevent you from even trying,
Are you sure about that? There are a lot of people who worry about going to the gym , concerned that everyone else will be staring at them. No one [except for the very rare hole] is bothered about what you're doing, they are just getting in with their workout. I'd expect swimming to be the same.
Our local pool has various sessions for beginners, both young and old. Normally the pool is separated into three lane sections, fast, medium, and splashing around.
Any 'stigma' you do perceive isn't worth bothering about - it's their problem and not yours.