Think of it as an ROI. The state invested heavily in you - schools, roads, doctors, police, fire brigades, clean air, etc. etc.
We expect repayment from all members of society in order to continue providing those benefits to them and those around them.
If, when you reach the age of maturity, you don't feel that you want to be part of the social contract, you have a few choices available.
1. Leave. Go off to Somalia or some other place where the state won't force you to contribute taxes.
2. Lobby your elected officials (or become elected) to explain why we should stop supporting society. Remember, all those libraries were erected because people in your society saw the value in collectivising to enhance public education.
3. Opt out of society as far as possible. Generate your own electricity and grow your own crops. Just don't impinge on our rights to live in a safe(r) environment.
But, by and large, the Libertarians I've encountered don't think like that. They have the misguided idea that all their successes are down to personal endeavour and that they shouldn't be responsible for contributing at all. They see tax as theft of their hard work, not as a investor extracting their fair share.
Of course, you're right that Libertarianism itself doesn't imply having no social conscience - but many of its adherents simply don't.