The pejorative use to refer to gay people wasn't the dominant use of the term until, well, quite late in the 20th Century, and the structure of the game makes it a whole lot less likely that the reference was a pejorative reference to a sexual minority rather than simply the odd man out.

It's kind of like if you someone argued that the “ball” in the name of association football was a reference to testicles and not the main piece of equipment in the game. Sure, the word also was used that way at the time, but it doesn't make as much sense in context as the alternative, and without some evidence beyond the mere existence of the sense of the word it's not a particularly compelling explanation.