Poker, while considered a game of skill, had a direct connection to gambling, and hence the negative connotation.
Poker isn't very interesting of it isn't played for stakes (whether it is money or something else) in fact what is the point to it, if you aren't playing for something?