It doesn't have to pass the Turing test to masquerade as you.

The John Legend voice for Google Assistant was apparently made without hours of recordings. Companies like https://lyrebird.ai/ do a basic job of copying your voice with about a minute worth of samples (more obviously helps).

I think it's entirely inevitable that at some point you will be able to customize the Google Assistant to use your own voice thanks to WaveNet and it's successors. Instead of the recognizable assistant making calls, every assistant will be different.