I always understand it to be missing the word actually:
"yes, actually no"
"no, actually yes"
"yes, actually no"
"no, actually yes"
Rephrasing it as yes, 'actually no' makes it sound like the speaker is indecisive which yeah nah does not communicate. It's a a social nicety to smooth over hurt feelings.
That of course is until you use it with an affirmative 'yeah nah yeah you're right' or similar.