Is diarization only possible with stereo audio at the moment in whisper? If the voices aren't left/right split I don't think one can get them separated yet.
Author of Wordcab-Transcribe here. We use faster-whisper + NeMo for diarization, if you want to take a look.
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en...