Yes, the selfish-minded would end up with more selfish-minded people, and they'd be confused why their "low trust society" became even more low trust overnight.
I suspect if you played this game, lots of tight-knit, high-cooperation groups would undertake coordinated campaigns to ensure the survival of their members by ensuring everyone votes red.