Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model | Hacker News Reader

Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model

google.github.io

339 pointsbackpropaganda84 comments

The stress and intonation examples are what puts this on the far side of the uncanny valley. I've not heard computer speech like this, where the voice seems understand the idea underneath the words.

It is without doubt very impressive, but on longer texts the lack of understanding will be apparent. Wrong semantic stresses, incorrect pronunciation of homographs, maybe inconsistent pronunciation of names, and so on.

Perfect speech synthesis from unannotated text will require much more complex system. It will need database of common knowledge in some form, ability to relate text to that database, ability to retain and use this semantic information to guide synthesis. That is, it should be a step closer to artificial general intelligence.

I'd say even in these samples there's still some errors. The intonation on "brown" in "quick brown fox" is still wrong. "AllHipHop" is not pronounced as a human would, though I'll grant you that's a made-up word so I wouldn't even expect a non-native human speaker to necessarily get that one right.

Still, the quality is such that I'm only observing this in the spirit of scientific analysis. It's some good quality.

I wonder whether it would have been more natural if it was written as "quick, brown fox" as would be customary English. Usually when adjectives don't have commas between them it's because the last adjective is coupled to the noun.

How is this different from the DeepMind's [0]WaveNet?

[0]https://deepmind.com/blog/wavenet-generative-model-raw-audio...

Quoting the paper, in the section related works:

"WaveNet (van den Oord et al., 2016) is a powerful generative model of audio. It works well for TTS, but is slow due to its sample-level autoregressive nature. It also requires conditioning on linguistic features from an existing TTS frontend, and thus is not end-to-end: it only replaces the vocoder and acoustic model. Another recently-developed neural model is DeepVoice (Arik et al., 2017), which replaces every component in a typical TTS pipeline by a corresponding neural network. However, each component is independently trained, and it’s nontrivial to change the system to train in an end-to-end fashion."

No source code, as usual.

How difficult would it be to duplicate these results with TensorFlow? Would something like this require more than the building blocks that TF and other toolkits provide? I have zero experience with machine learning, so I'm just curious.

I think implementation can be done with TensorFlow but hardest part is to provide relevant training set to model.

> How difficult

Very difficult:

1.Machine learning is pretty data dependent, and make those datasets are very expensive. Google is not likely to give them away for free, because it is their competitive advantage.

2.The infrastructure to train those models are hard to get outside of Google. Pretty sure it is 10s or 100s of GPUs, with Infinity Band connected PS server, running for days and weeks.

Even with source code published, people will still have to scratch their head to duplicate Google's performance. Until the day, some equivalent organization as GNU that democratize data access to the public and some mighty algorithm being discovered dramatically reduced the computational requirement for training those models, Google succeeds by just being Google is unlikely going to change.

1. If you can pay for around 24.6 hours of VA speech data, you can get enough data to run this process with the same quality that Google presented. (that's from the "Experiments section") Not cheap (definitely not free, especially considering the amount of quality control you have to apply), but not expensive either.

2. You can rent out a 96GB GDDR5 GPU instance from Google's cloud for pretty cheap. (https://cloud.google.com/compute/docs/gpus/) I don't think you need anything more powerful than that (but feel free to prove me wrong).

I think your last paragraph is totally misguided/uninformed. You can download models for cheap/free (for non-commercial/edu use) from UPenn (https://www.ldc.upenn.edu/language-resources/data/obtaining). People don't give away models for free with 0 strings attached because they're a pain to make.

And if you want something you can run on a home computer for cheap/free, you can try DeepSpeech: https://github.com/mozilla/DeepSpeech. All you need is an Nvidia GPU.

How about feeding it several audiobooks read by a single narrator, coupled with the books in text? Cost would be < $100. There could be legal problems if you tried to sell the resulting voice, but as proof of concept, wouldn't this work?

1.What is your estimate then? Hundreds Dollars? Thousands? 10 Thousands? Surely it might fall into the later two category, since paying a professional speaker to sit and work for 26.4 hours is already over 1000 dollars, if you assume 40$ per hour wage.

2.96GB GDDR5 instances on GCE costs 4166.4 dollars per month. Though it is within affordable range, but definitely not CHEAP. I don't know whether this is powerful or not, but Google used 96 GPUs for their GNMT work. Thus, I don't think I have the confidence to say a 4-GPU machine is all you need, and it will surely cause much more if you go beyond that.

You're describing the future : 100% capitalistic society... I'm not sure I like it...

Right, even a good diphone voice needs lots of data. And I noticed they trained it with the existing Google Home voice actress, from whom they must already have many, many hours of recordings. I was mostly asking about the model itself; whether you could download TensorFlow and put one together based on this paper alone.

I see your points. But it is related. Even if u get what you think the paper describes, it is hard to know whether you did it right or not, because you cannot replicate the result easily. This happens in a lot of CV papers already, where people reimplement the model, but it never get as good as the paper demonstrated

But, you have a very good idea. Since it is Google Home, will it be possible that some people just buy hundreds of them, and infinitely ask them question to gather the training data? That will be interesting to see.

They mention implementing the model in TensorFlow, so it should absolutely be possible. (Unless they are using a special Google-only version.) One thing that might make reproduction difficult is that they don't describe their initialization procedure. But maybe it just doesn't matter.

Initialization is random according to abstract.

  > Given <text, audio> pairs, the model can be trained completely from scratch with random initialization.

Yes, but random according to what distribution?

No source code, as usual.

So why is it on Github?

To make the voice samples easily available. Preprint archives and scientific journals don't usually allow you to embed audio into your papers.

https://github.com/google/tacotron

Still no source code...

Easy way to publish?

Feels like we're getting breakthroughs every few months now. How close are we to the point yet that a mobile phone can generate speech in close to realtime that is reasonably pleasant to listen to for long periods?

Audio book piracy is gonna skyrocket when you can generate a pleasant track from text.

I just think there are huge opportunities for new types of UI once it works. Any time I'm out and about with headphones, I want something like a custom radio station, mixing up my music and podcasts, dictating messages and highlights from social media or news sites I like. I'd even accept short adverts if it was genuinely relevant and location aware. Even when I'm not travelling, I suspect it'd be a cool interface to have at work. If something can check in from time to time between music tracks, giving me a quick run down of anything interesting that's happening (even plugins for the build server etc), it will cut down on my temptation to open it up in a browser tab.

I've wanted to build this app just for myself for years, but it's only very recently that it seems the state of the art is close enough for it to be palatable to users.

You can model your face and your voice in future RPG games.

Huh? I would think, if anything, it would go down if, as an alternative to pirating an audio book, one can download the text from Gutenberg.org or pirate the _text_, and generate a decent audio book from it.

True, technically it's not "audiobook piracy", but I used that as shorthand for recording and distributing an audio version of a written book, which courts will surely find to be a form of piracy.

I think that's what parent meant, it was just phrased oddly.

can we eventually apply style transfer to speech? I'd like to hear all audio narration in the voice of David Attenborough.

Maybe not yet, but I am sure there will be speaker cloning extension soon, like photo and painting style transfer.

Edit: It would seem they only need 25 hours of audio to train the model, so it would probably be easy to find or make alternate datasets.

> We train Tacotron on an internal North American English dataset, which contains about 24.6 hours of speech data spoken by a professional female speaker

It's not really style transfer, but for a new speaker model, you just need to train each speaker with a dataset of 25 hours audio with time matched accurate transcriptions.

In the case of David Attenborough, I'm pretty sure that amount of data is available with subtitled BBC content.

Oh I doubt that will be necessary. I bet you could find a useful "feature vector" by repeating the process on about 10-20 new voices - then extract a set of words (let's pessimistically say 100) that activate these features the most - then have a person retrain with 5 examples of each of these 100 words (an hour, at the most).

I honestly wouldn't be surprised if you could find a really feature rich sentence (say 5 words long) that you could use to crack pretty much all voice-activated biometric password systems.

Is voice-activated password a thing? I mean, I only saw that in sci-fi movies and never in real life.

Yes, voice biometrics are used in telephone banking for instance

yep, I'm sure there would be a lot of similarity between models that could be used to reduce training time

Adobe VoCo does something like that. Demo: https://www.youtube.com/watch?v=I3l4XLZ59iw

It was pointed out the last time I posted this that the demo is "fake" at least in as much as the inserted words aren't being synthesized but rather copy-pasted with existing intonation intact.

That's pretty cool; any info on a release date?

My understanding is that while a number of people have tried, we currently have no particularly viable approaches for 'style transfer' analogues on sequence data, including text and sound/speech. Samples from past work are pretty bad. But what you could possibly do is roundtrip audio to text and then text (with conditioning like speaker metadata) to speech, both of which are doable. But directly transforming, not yet.

You mean, like they did pre-2000 [1]? Don't know how good this would sound today, but nevertheless interesting.

[1] http://www.washingtonpost.com/wp-srv/national/dotmil/arkin02...

I was wondering the same thing! I think, in the future this might be a problem for national security

Comedy option: Gilbert Gottfried.

Or Stephen Fry

Or Bill Kurtis? :)

Being able to pronounce Taleb Kweli is blowing my mind. Absolutely amazing, can't wait to see where this goes.

It doesn't, it's Tah-leb Kwah-lee, that sounds more Ta-Leb Kwell-ee

But damn, the rest of it is astounding

It sounds good to me. Not sure why you'd want to pronounce it as Tah-leb Kwah-lee?

If anything, it sounds like "quiley".

Not trying to sound snarky here, but that's how he actually pronounces his name.

Paper link here: https://arxiv.org/abs/1703.10135

I'd love to see it take on this poem: http://www.thepoke.co.uk/2011/12/23/english-pronunciation/

That was a great poem, storying that one away for later. It truly shows the complexity of the English language.

I'm totally new to tts, so please forgive the questions:

* Is this open source?

* What hardware would one need to run this?

* If specialist hardware is required (e.g. reasonably high end GPU) would that simply be for building and training the system or its operation?

From what I understand looking through the paper, this is only a research paper, without any source code. So right now it's not possible to say what the hardware requirement might be.

Thank you. It would have made more sense to me if the source code to generate the output examples which they posted. Oh well.

I would like an on-line version that can generate mp3 from supplied text. In English using a British accent. Female.

I have my reasons.

couple this with a system that transforms pictures of people into nude pictures of the same people. win $$$. watch civilization crumble to dust.

There's a reddit forum that has pictures of on/off. Maybe given enough of those as the training data, one could create such a thing.

Make it also work in realtime over video and add VR capabilities. Win lots of money. Watch civilization crumble to dust.

Scottish, Geordie, Liverpudlian, Yorkshire, Birmingham, Welsh, Somerset, Cornwall, East Angular, Estuary, Cockney, or from somewhere else?

Posh!

Question: why do people generate waveforms by neural networks?

Would it be easier for a neural network to "control" an emulated throat/mouth and train against the sound output?

Emitting and post-processing waveforms entirely in neural networks seems like putting a lot of responsibilities into neural networks.

The reason is, the only speech data that is available in high quality and large quantities is in audio form. There has been research for decades on measuring or estimating the properties of the speech organ (oral cavity geometry etc), but that is largely an unsolved model inversion problem. Existing physical speech synthesis models are pretty crude and sound quite robotic.

I like that you mentioned "inversion".

ML is strongly related to solving inverse problems but it does not tend to be emphasised enough, in my opinion.

In reality, speech organs (a relatively low dimensional system) generate high dimensional data (speech). This is an example of the so called curse of dimensionality.

What ML does is to invert this process and create a model of the generation process.

My favorite example is quantum mechanics. Physicists made lot of measurements and created a model (quantum mechanics) that reproduces those measurements (and even works on test data, not only on training data).

This is an example of a solution of an inverse problem. ML does the same.

So in one way or an other, every working TTS has a model that accurately describes the speech organs and how the brain uses them to create speech from text.

> Existing physical speech synthesis models are pretty crude and sound quite robotic.

Could you provide some links/publications please?

I'd really love this to mature completely and applied to Kindle Whispersync books. I've found that some readers of audiobooks make the book hard to comprehend or even distracting, especially if the sex of the person does not match the author or even remotely the sound of the author who I may be very familiar with from video interviews and what not. Ideally every author would be a great reader and create recordings for their own content, but if some synthetic means could achieve an approximate result that would be fantastic. Even better people could choose from a flavor of voices and settle upon the one that works for them.

Unless all these fancy new text to speech engines are released as actual software instead of cloud services they're worthless. And we all know that isn't going to happen.

How is research worthless?

Worthless for who? Millions of people use these engines every day.

I've been a long-time user for OS X's built in text-to-speech [0].

IMO, it actually performs pretty robustly on these examples. Is Apple still using just diphones, or are they post-processing in some way?

[0]: https://en.wikipedia.org/wiki/PlainTalk#Text-to-speech_in_Ma...

I'm using Alex to read articles and comments. I love the Alex voice but it sometimes makes stupid mistakes.

I hope we will have decent TTS for linux too.

Fyi they added newer, (imo) better quality voices since alex debuted in 10.5. My personal favorite is Kate (english UK voice) and Tessa (english - south africa)

This is really awesome! Anyone has any resources on learning TTS synthesis?

Jurafsky & Martin is the canonical textbook but it's pre-deep learning. Check out the wavenet paper[0], and Alan Black's Festvox[1].

[0] https://deepmind.com/blog/wavenet-generative-model-raw-audio... [1] http://www.festvox.org/

These new models are replacing almost everything that's been developed for TTS in the last 30 years with neural nets. So you don't need to study TTS anymore. Study neural nets and then read this paper and you will understand the state of the art in TTS.

This is complete nonsense. You need to understand the problem you are addressing even if you plan to use deep learning methods for them.

Saying you don't need to know anything about TTS is like developing a self-driving car and saying you don't need to know anything about the rules of the road. For one thing, how are you supposed to know when you got it wrong? And what things it is important to get right?

I've used AT&T's Natural Voices SDK about 10-15 years ago? It allowed you to create your own voices, but it's a hell of a lot of work.

this book is a classic in the field :

http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf

This model sounds so cool. Any idea of a high level software package where you can feed pairs of <text, audio> and build a TTS model as a result?

This research just came out, quite literally, like today.

There will be a tensorflow implementation tomorrow though.

Do you have an update on this topic?

What would happen if trained on 2 or 3 voices for the same amount of time?

The model is smart enough to not mix them together in the same sample. Different samples would have different voices, unless you put in a voice identifier as input to the model. Then you can control which voice to generate with.

Where is the code?

Reply on news.ycombinator.com