Hacker News Reader

Comment by hacker_9 | Hacker News Reader

Additionally when we speak we don't just 'replay' words; we add emotions and so on. Reminds me of this video [1] analysing how the actor Anthony Hopkins converts his lines to speech for his scenes.

[1] https://www.youtube.com/watch?v=4kSGkGKwp9U

View on HN

The spoken word has so many nuances. Cadence, inflexion, tone, and then the facial performances are never captured by pure text to speech. You'd almost have to be able to set boundaries for seriousness, playfulness, innuendo, indifference, sarcasm etc because none of that is conveyed in the literal text without additional descriptions or contextual analysis of the conversation.

It'd be interesting to take the Joint Many-Tasks approach here, training some of an RNN's layers on text-to-speech, and then another set of layers on sentiment analysis of speech-audio, where the error of the sentiment layer can backpropagate into the TTS layer.

I guess you could do something like vocaloid does to better "shape" the sounds of each word. I guess its pretty tough to do that automatically though, since words by themselves don't convey the emotions that they should be portrayed with.

I mean, a word or phrase could very much change meaning depending on how its spoken (excitement, disgust, fear, sarcasm etc) and even if the context is available in the text (as is often the case), its rather difficult to machine-detect, at least, for now.

Reply on news.ycombinator.com