That's a specious argument. For example, I might not have children, does that mean I shouldn't pay school tax? No, we all pay the tax, whether I directly benefit from them or not. It's a social contract.
Allowing for special classes is the problem, not a solution. It is the collective good we are striving for. Otherwise, we end up in a situation of the tragedy of the commons, and no one wins.
Perhaps, instead, we should also focus on extending life expectancies for African American men, rather than allowing them to not contribute to Social Security.