I then transfer learned for each of these speakers. Some speakers have as little as 40 minutes of data, others have up to five hours. The resulting quality isn't strictly a function of the amount of training data, though more typically helps. It's also important to have high fidelity text transcriptions free of errors.

The transfer learning runs vary between six hours and thirty six hours.

I'm using 8xV100 instances to train glow-tts and 2x1080Ti to train melgan. I'm continuously training melgan in the background and simply adding more training data. The same model works for all speakers.