Jared Diamond is a pretty big proponent of environmental determinism. This can be somewhat obscured in GG&S itself, but his follow-on book Collapse is a pretty naked advertisement for environmental determinism, and he harps on a lot about it in interviews and the like. In fact, the introduction to Why Nations Fail (if memory serves correctly) includes some discussion by people who saw the book before it was published, and one of those discussions was Diamond, who complained that the book rejected environmental determinism as a major factor (to which the authors responded "the evidence just doesn't hold up, what more do you want us to say?").

A lot of the complaints about GG&S are to some degree a complaint about Diamond in general being mapped on to the least offensive of his works, and if that is the only exposure you have to Diamond, it can be seem like the complaints are coming out of left field.