51 karma · joined March 1, 2019
Style transfer is trickier to do with speech than images! One significant issue is the lack of a good "content" versus "style" distinction. In images you can get great results by calling the higher-level features of an object classifier network "content" and holding that constant. Some people have tried this for audio with e.g. a phoneme classifier, but there are additional characteristics (such as inflection) that relate the emotional content of speech which wouldn't be held constant.
Another issue is that much of the speech classification work is done in spectrogram (or with further processing MFCC) space, which lets you treat audio similar to images and leverage a bunch of technology that we have for classifying those. But for synthesizing speech, spectrograms aren't a fantastic representation, because small errors in spectrogram space can translate into large errors in the waveform which are very clearly audible, and humans in general are pretty sensitive to audio errors. There are cool neural spectrogram inversion methods out there which can help, but those should still be trained to be robust to the kinds of errors that a style transfer algorithm would make, so it's still pretty tricky.
My company, Modulate, is building speech style transfer tech; and we've found a lot more success with adversarial methods on raw audio synthesis, where the adversary forces the generator to produce plausible speech from the target speaker!
One of the coolest parts of the kind of BMI research in this article, to me, is the potential to buy back some latency margin for speech conversion! If you're working on already-produced speech, there are super tight latency requirements if you want to hear your own speech in the converted voice - over 20-30ms for the entire audio loop, and you start to get echo-like feedback that makes speaking difficult. Even without looping back, you don't want more than 100-200ms of latency in a conversation before it starts impeding the flow of dialogue. This means your style transfer algorithm gets almost no future context, and limits the kinds of manipulations that you can do (not to mention the size of the network that you can do them with, depending on available compute power!).
That said, there _are_ some interesting things to be careful around, even for changing your gender, or age, or other basic variables around your voice. We're mostly worried about what impact this would have for communities built around these kinds of commonalities: for example, is it okay for a child to masquerade as an adult in an adults-only social group? I don't think there's a clear answer to all of those situations - but until we see more use of realistic voice skins in the real world, we're playing it safe and building in these kinds of tools!
It's a really interesting technical challenge. On the one hand, changing your timbre to another human voice is much more complicated than basic pitch shifting, so we ended up using deep neural networks for a kind of simultaneous speech-recognition and speech-synthesis approach (though training this system to preserve e.g. the emotional complexity of your input speech while still changing out the timbre convincingly is difficult - we use adversarial training on the raw audio waveform, which is powerful but also pretty much unknown territory compared to images).
On the other hand, it's important to run with very low latency on your device while you're playing, which means that we can't simply "throw the biggest network we can" at the problem. So we have a tradeoff between model latency, which is easy to characterize, and audio quality / voice skin plausibility, which is pretty ambiguous and subjective.
Finally, as this kind of tech improves the potential for misuse becomes an important problem, so we need to build in protections (like watermarking the audio) that can help prevent fraud while not compromising the speed of the algorithm or the quality of the output audio.
Modulate is developing real-time voice skins to allow gamers and social app users to digitally customize their voice as they speak! Sound like the character you're playing; your favorite celebrity; or design a whole new voice unique to your online persona. (Try it at https://modulate.ai for yourself!)
We're looking for developers to help us deliver our tech to our pilot customers. Our machine learning engineer will work directly with our CTO to continue researching new techniques to improve the quality and speed of our voice skins, while the core engineer will help drive our work on integration with a variety of platforms while maintaining real-time speeds.
We use tensorflow and python to prototype and C and C++ in production.
Learn more at https://modulate.ai/careers, or email careers@modulate.ai to apply.