> How come nobody has reverse engineered the iMessage protocol properly?
People have, but changes have been made where the barrier to entry on the crypto/client verification side is sufficiently high a barrier to make it not practical. See next answer for more context.
> Is Apple using any hardware stuff to prevent this?
Yes.
> I think an unofficial iMessage client appeared a few years ago, but it was very closed source and I don't think it worked for very long
The iMessage team scrambled on its release to block it as it leveraged a severe security breach.