All messages, sent while a person is offline. It is bad, but not nearly as bad as "all messages"
Furthermore, as soon as the sender attempts to deliver another message to the recipient, they would get another notification (because the encryption key changed back to the real key); alternatively the attacker could continue blocking (and reading) messages to the recipient, but the lack of delivery would be noticeable.
You could escalate it into a MITM rather easily, though, by attacking both ends; but again, a key change notification should be displayed to both parties.
Assuming the closed sourced app works as advertised, obviously.