Of course it's up for debate. Debate is what gave the notion of "inherent worth"[1] intellectual and popular credence in the first place. You forget that human beings were historically categorized according to a chain of being with the clergy and royalty (rather conveniently) at the top. One's worth to $DEITY and the world was determined by the height of the seat one sat on. This arrangement of affairs was treated as an unquestionable given for thousands of years in civilizations across the world. To suggest otherwise would have been treated as insanity or denounced as heresy.
The existence of Inherent worth thrives upon the fact that those who debate it depend on it. To suggest it's beyond questioning turns the idea into an article of faith.
> Are cultures that decide you have no worth as a human being based on your skin color, or religion, or caste, or last name, equal to those that don't?
Until midway through the 20th century in certain parts of the world, this was accepted as fact. In other parts of the world, such as sub-Saharan Africa of all places, it's still the case today.
[1] A more rigorous term for what you're referring to is self-sovereignty. Strictly speaking, the term "inherent worth" is a contradiction in terms.