Depends on your threat analysis. If you're not considering an active attacker who has an ability to seamlessly fake voice and/or face to be a realistic attack scenario (e.g. you're guarding against mass surveillance, but not a prepared targeted attack - i.e. you don't find it plausible that someone has prepared actors - human or machine - with quality deepfakes made specially for you and your peer) and you know the other party personally, you can verify in-band, relying on natural biometrics aka your knowledge of one's voice and face. If you do - certainly the verification must be done out-of-band.