The reason iMessage works well is because it's integrated tightly into the OS. So for an open-source alternative to be competitive, it would have to run on... an open mobile OS, which would in turn require open hardware.
I don't see the need for absolutely open hardware for an alternative messaging system. Sure, they aren't FLOSS solutions, but they are much less closed than iMessage and are cross-platform.
Note: Hangouts is an abomination.
I'm not 100% clear on this: why can't there be an open mobile OS that runs on (bootrom-exploited) iOS devices?
Sure, you can use proprietary hardware (iPhone) and try to make it run different software, but your luck getting around their protection systems may vary (and it may or may not be illegal, depending on how much you care).
Hell, it can be difficult to get an unsupported version of Android to work without major bugs on an Android phone.
They had to rewrite the drivers, of course.