The use of baby as lover is well known, but not when phrased like this. I think its because 'baby' needs to have ownership attributed to it to imply partner.
I am an native english speaker that moved to Sweden. I am currently learning swedish and the swedish language doesn't use commas in the same way. I think a comma, even in just the lyrics (or a pause in the song), would have made things clearer.