Hacker News Reader

Comment by derefr | Hacker News Reader

One thing I always still notice about these "lifelike" speech models is that they still have random pitch variation that wouldn't be present from a real speaker—sort of a "warbling", similar to audio heavily compressed by some cellular realtime audio codecs.

I assume this is due to random variation in the pitches the speakers in the training data used to say given phonemes. Would this, then, imply that speech models for tonal languages like Mandarin don't share this problem, since people are speaking with prescribed pitches more often?

View on HN

I don't know about tonal languages. But you're definitely right about the artifacts of concatenative speech synthesis.

It's worth noting that purely synthetic speech, while lacking the superficial lifelike quality, doesn't have these little inconsistencies. Perhaps that's one reason why many blind people, particularly power users, prefer pure synthetic text-to-speech engines, such as ETI-Eloquence (commercial) and eSpeak (open source). These systems especially sound better at high speeds than the newer "lifelike" ones.

Just for fun, here's a side-by-side comparison between a concatenative system, IVONA (acquired by Amazon):

http://mwcampbell.us/tmp/derefr_comment_ivona.mp3

And a purely synthetic system, Eloquence:

http://mwcampbell.us/tmp/derefr_comment_eloquence.mp3

The latter is the text-to-speech engine I use every day, and that clip was synthesized at the speaking rate I normally use.

Edit: It appears the word I was looking for when I said "purely synthetic" is "parametric".

How do you feel about the WaveNet results, by comparison?

• https://storage.googleapis.com/deepmind-media/pixie/us-engli...

Those phrases sound very natural indeed. But are they carefully selected demo phrases, for which the system might have been tuned? That's always a danger with marketing demos. So I have two questions:

1. How does that engine sound when reading arbitrary text at high speed?

2. Can I run it on my own computer, so I can have low latency in my screen reader? (That, of course, is a problem with Amazon Polly as well.)

Additionally when we speak we don't just 'replay' words; we add emotions and so on. Reminds me of this video [1] analysing how the actor Anthony Hopkins converts his lines to speech for his scenes.

[1] https://www.youtube.com/watch?v=4kSGkGKwp9U

The spoken word has so many nuances. Cadence, inflexion, tone, and then the facial performances are never captured by pure text to speech. You'd almost have to be able to set boundaries for seriousness, playfulness, innuendo, indifference, sarcasm etc because none of that is conveyed in the literal text without additional descriptions or contextual analysis of the conversation.

It'd be interesting to take the Joint Many-Tasks approach here, training some of an RNN's layers on text-to-speech, and then another set of layers on sentiment analysis of speech-audio, where the error of the sentiment layer can backpropagate into the TTS layer.

I guess you could do something like vocaloid does to better "shape" the sounds of each word. I guess its pretty tough to do that automatically though, since words by themselves don't convey the emotions that they should be portrayed with.

I mean, a word or phrase could very much change meaning depending on how its spoken (excitement, disgust, fear, sarcasm etc) and even if the context is available in the text (as is often the case), its rather difficult to machine-detect, at least, for now.

Southerns have an accent. New Jerseyans have an accent. British have an accent. Can't computers have an accent, too?

Wouldn't we describe what TTS systems do as more of a speech impediment? Sure they use a particular accent, usually that of a standard prestige dialect. But it's not the accent that's weird.

Sheer speculation, but I wonder if that could be related to the "checkerboarding" artifacts you almost always see in AI-generated images?

Here's a good writeup: http://distill.pub/2016/deconv-checkerboard/

Re: the checker-boarding artifacts, that sounds interestingly like deconvolutional NNs treat their samples, at each deconv layer, as independent, rather than taking the Rule of Addition of probability-weights (P(A ∪ B) = P(A) + P(B) - P(A ∩ B)) into account. (Would a model that did do this then be a "Bayesian Deconvolutional Neural Network"? Does that even make sense?)

I doubt this would result in audible artifacts at the audio level, though. The visual checkerboarding appears at the 1px detail-level of the output image—this would translate more to minor fluctuations along an audio sample's Nyquist frequency, which is effectively undetectable if the output sampling rate is of the usual kind (44.1kHz).

Chinese (Mandarin) text to speech still shares the problem of not sounding natural (sounds robotic), but it is a little better than English.

Example (click on the blue speaker icon): http://dict.baidu.com/s?wd=%E7%88%B1%E5%B1%8B%E5%8F%8A%E4%B9...

I'm not sure about Mandarin, but this is definitely true for Japanese. Near-human Japanese text-to-speech has been around a very long time.

So good they can even sing, and so well that they create entirely virtual pop stars: https://en.wikipedia.org/wiki/Vocaloid

Vocaloid still sounds robotic (at least, to me). It just so happens when you make autotune/melodyne'd pop songs, it really doesn't matter.

Vocaloid isn't automated though, it's basically an instrument that humans "play". The software has specific controls for pitch, aspiration, etc

True, but Japanese isn't really a tonal language, so it doesn't really evidence the parent's point.

Japanese isn't technically "tonal" but does have "prescribed pitches" which certainly affects the ease of text-to-speech.

I always wonder why they don't program a second pass over the output with a model, say, trained on radio conversations. That second model just for audio 'quality'.

If you grab the training set for CMU-Sphinx, you'll see that the voice actors providing base input are trained to speak that way.

At a secondary level, I imagine there's a sort of pitch trajectory for a sentence, and if a word or syllable ends too far off that trajectory, the return to the pitch trajectory is fast enough to be easily detectable.

I was wondering why it sometimes sounded like bad compression artifacts! Thanks.

Reply on news.ycombinator.com