Something like a CycleGAN and/ or weight sharing would do the trick