If society is unwilling to take responsibility, then society has no right to expect obedience.
Sure, society can always compel and force somebody to obedience. That's why a society usually tries to monopolize the exertion of power through law enforcement. That might even give society the power to expel a homeless mother along with her children from a house that's been empty for 35 years.
My argument is not about ability but about morality.