Furthermore, as soon as the sender attempts to deliver another message to the recipient, they would get another notification (because the encryption key changed back to the real key); alternatively the attacker could continue blocking (and reading) messages to the recipient, but the lack of delivery would be noticeable.
You could escalate it into a MITM rather easily, though, by attacking both ends; but again, a key change notification should be displayed to both parties.
Assuming the closed sourced app works as advertised, obviously.
Unless the key-change forces the user to be using an insecure key-pair, but is that actually happening?
It is hard to check what WhatsApp does, but in Signal it is not the server, but a recipient who sends delivery receipt. WhatsApp then has to either recognize encrypted receipts or allow only one-way conversation during attack. Carrying out the whole attack just to decrypt "hi, are you here?" is not really interesting.