There are multiple factors and I’m not sure what the exact cause is either. The relative difficulty to build a playable bot is one. A Go game being relatively long and taxing may be another. Go also traditionally has a reputation of being a game for bureaucrats and intellectuals, while Shogi being a more people’s game. The obvious parallel between the battlefield also makes Shogi more popular among samurais, who yakuzas draw a lot of inspiration from.

In general I think the two are of comparable popularity in Japan, but I wouldn’t be surprised if real yakuza people do like and play Shogi much more than Go.

Decent relative to what? These games include various card games too which in theory are even harder for AIs to beat pros at. I'd be surprised if the other minigame engines are actually decent to a skilled amateur on their hardest difficulty. The AI complexity is also irrelevant to many of these games being playable with a friend in person or online.

The length of the game is also something easily tweaked, just default the board size to 9x9 and offer 13x13 and 19x19.