In general I think it is silly that voice cloning research has focused so much (exclusively?) on cloning voices from just a few seconds of audio. It puts a pretty low ceiling on quality. Many nuances of a person's communication style will not be contained in such a small amount of data. Sure you can match their pitch and timbre, but voice cloning should be more than that.
You don't have to suppose anything: it is actually settled law that its bad to just willy-nilly use people's voices if you feel like it, even if its just a sound-alike!
Not your voice, not your IP, not your right.
> Voice conversion refers to altering the style of a speech signal while preserving its linguistic content. While style encompasses many aspects of speech, such as emotion, prosody, accent, and whispering, in this work we focus on the conversion of speaker timbre only while keeping the linguistic and para-linguistic information unchanged.