No, they don't need a pre-trained model, they start from scratch. Also it's only the approach which is common between modalities, they train them separately.
"Our work does not perform multimodal training but aims to unifiy (sic) the learning objective for self-supervised learning in different modalities."