> Grammar is always the toughest.
I think this depends somewhat on the language. For some, grammar is the first major bottleneck to get to a usable beginner level, but for others, it's pronunciation. For example, Danish grammar is relatively simple (which is one reason Google Translate is very good translating it), but it's quite difficult for non-native speakers to pronounce it intelligibly. I've also heard second-hand that pronunciation is a bigger problem than grammar for beginning Mandarin Chinese speakers.