Masks are needed to protect not the wearer, but those around him. Same reason why doctors wear masks when performing surgeries.
It's been proved that masks help the wearer too.
It's also being theorised that masks, due to not being perfect, is giving immunity by exposing one to lower viral loads and leading to very mild or asymptomatic disease.
As an aside, if you have a valved N95, just put some duct tape over the valve and voila, unvalved N95.
It's an interesting idea "everyone else is responsible for avoiding the invisible thing you're spreading around", interesting in a stupid way, but interesting.