There's no need to round-trip through text, you "just" need to train an embedding space that captures both domains.