I think it's quite simple - subjective experience. Under the assumption that others experience things (much like you do) you can just ask or infer what would they prefer, and what would cause pain. You can be wrong about what would they prefer (and they can be wrong), but it doesn't mean there's no coherent morality hiding under such mistakes.
And I don't think that it's much to assume. I think it's no more that the general assumption that "causes have effects", or that there are any invariants we can attribute to the universe (like laws of physics).
Note that this notion of morality extends beyond humans - for example to animals and potentially AI systems.