Google+ was a social network. For a social network to have any value, you need your friends to be on it as well. By making it invite-only and throttling how many people could join, they guaranteed that most of your friends weren't there and COULDN'T be there. There were so many memes being made of Google+ being described as this amazing party you got invited to, only to get there and find there's nobody there.
GMail being invite-only at first wasn't a problem because using GMail didn't require everybody you sent/received e-mails to/from to be on GMail as well.