If they are only using GCM as a queue (and the messages are themselves encrypted) I don't understand what the problem is.
They could use anyone for that functionality. Even if the messages are given to an "adversary" what can they really get from that? Your phone app contacted the signal servers. That's really it.