My understanding: If you verify the safety numbers in person, then I believe you can be confident that it's E2E encrypted for that conversation. If the safety numbers are different, then there could be a nefarious actor listening in.
Someone please correct me if I'm wrong.
Edit: That being said, I believe they could still record IPs, as well as the destination and timestamps of each message.