Basically what we as humans do every day.
And I know that as well as I know anything I'm not an expert in -- from my own cursory review and stated opinions of people who are experts (again, as far as I know).
In other words, it would be surprising otherwise?
Our morals/desires depend on the way we are built and they dictate the 'oughts', thus providing a link from 'is'.
And of course it is possible that we can never have enough information to decompile our own values because it is lost in compression, so we'll be stuck with dealing with a black box. But this is a technical limitation, not a philosophical one, and for a simpler system (for example a AGI if one can be built understandably) we would be able to derive its 'oughts' from what it 'is'.
Just because something is intuitive doesn’t make it true.
Empathy doesn't have much to do with morals, it is closer to a mechanism of knowing what the other entity feels. This is useful in antagonistic scenarios as well as in cooperative ones. What you use it for is entirely up to you (and your built-in morals).
I agree that entire society can be wrong, even if all other societies of a given species share that opinion at the time. Where relativism would come in is where there is a major difference in mind architecture.
[0] - https://www.lesswrong.com/posts/HawFh7RvDM4RyoJ2d/three-worl...
We also know it breaks down in hypotheticals. In that example about a world of psychopaths, most would agree right and wrong remain constant. But a different hypothetical, where god and sacred texts say the exact opposite of what they do, we would not see as many people saying their god would now be wrong.
Furthermore, we do know somethings about "ought" like that it is not entirely relative, remains constant across world changes and, is vital to pursue. But we don't know were it comes from and why it varies slightly among cultures. So while we know right from wrong and that such distinctions are vital, we don't know how we know this.
It might be that what I am describing is not quite what is meant by "ought". But there are other vital "self evident" concepts that needed adjustment to their definitions yet those adjustments neither destroy the entire concept nor even make any practical difference.
Time, for example, is not quite what people intuit and heat seems self evidently entirely different than light yet is not. Yet those facts make no difference in every day life. I expect a better understanding of the origin of ethics will follow that pattern.
Society would then promptly adjust the laws to follow suit (it is a true source of morality after all). Would you then consider murdering 14th stranger you meet today if that brings you some benefit (outweighing the risk, etc)?
If not, why not?
(Any similarities to religious texts is intentional)
Morality only seems universal in our society because we're all human, and most of our morality is derived from emotional responses preprogrammed into the human brain as a product of evolution. To evaluate the morality of more complex actions, we build architectures of logic on top of those emotional responses, and there's plenty of room to argue about the best way to do so. But fundamentally, murder is bad because it feels bad, and it feels bad because we evolved to live in groups; that's all there is to it. Any arguments about how forbidding murder creates a more stable society are fundamentally secondary. At best they're explanations for why we evolved this way, and/or reasons to strengthen a pre-existing moral precept. They can also encourage us to suppress emotional responses inclining us toward murder, because those exist too... but the effect of the suppression is limited, and in practice people sometimes commit murder anyway.
Importantly, though, the fact that my morality is in some sense 'arbitrary' does not mean I have to tolerate other arbitrary choices of morality. If some person or AI is strongly driven to kill lots of people or fill the universe with paperclips, I'm not sure I could argue that my morality is inherently superior. But I'm still going to try to stop them, because that's what my morality tells me to do.
Internal source like you described is also objective, it is shared among all humans and based on objective realities of evolution and history.
I bet psychopaths share significant portion of it with neurotypical humans as well -- they just have different weights among multiple contradicting heuristics and/or certain mechanisms working slightly differently.
but how?
I think you might not appreciate how vexing the is/ought question is. murder is something that almost everyone agrees is wrong. most people couldn't exactly articulate why they think this, but I suspect the reason is that obviously causes ripples of pain, distress, and instability through a community. we may even be biologically conditioned against murdering our own kind.
so clearly, if one wants to remain part of society, not cause unnecessary harm, and not experience guilt, one ought not commit the act of murder. but this is still not a universal moral claim; it depends on the desires of the agent. you are conflating practicality with the philosophical notion of truth. these philosophers are not interested in "a sense of ought"; they are looking for a rigorous method of constructing "ought" from what "is". that is the problem.
I think it's quite simple - subjective experience. Under the assumption that others experience things (much like you do) you can just ask or infer what would they prefer, and what would cause pain. You can be wrong about what would they prefer (and they can be wrong), but it doesn't mean there's no coherent morality hiding under such mistakes.
And I don't think that it's much to assume. I think it's no more that the general assumption that "causes have effects", or that there are any invariants we can attribute to the universe (like laws of physics).
Note that this notion of morality extends beyond humans - for example to animals and potentially AI systems.
I'm not saying this is some huge conundrum that we need to solve before we decide to go get breakfast without killing everyone we see along the way, but the problem as stated is a hard one.
And I think all moral systems are essentially built around avoiding pain. Even if you allow for supernatural beings and follow the tracks of some religion - it's still a concern over well being, except in the after-life (for which we may or may not have much evidence for ...).