Right away I was presented with suggestions for people on Facebook that I might know, and they were obviously scraped from my GMail account.
So, as far as I could tell, Facebook got access to my GMail account without ever asking me anything about it.
I'm pretty sure they do save email data of non registered "users" because I was invited to facebook a few times. There was nothing strange on the first invite, but every consecutive invite had other additional friends of mine, who are too "waiting for me" to join.
Actually the one who invited me and "caused" me to join was someone I didn't really know, but met once at an event. This person was extremely unlikely to be friends with any of my real friends.
But still, I got suggestions for my actual friends right away, so I still can't see any other way than for Facebook to actually have had access to my GMail account prior to me joining.