The potential use of TPUs for training is very exciting. They say that they train floating point, but I don't see any indication to the FP precision they're capable of; perhaps I'm missing it. At any rate, I'm really excited to see resources being piled into their Cloud ML Engine product at this high rate.

I've made this comment in a couple of other threads, which subsequently veered off into other territory, so forgive the repetition, but it's a really interesting topic to me. The open-source distributed tensorflow stuff is pretty nice, but it still requires a huge amount of hand coding and tuning the machinery, reminding me quite a lot of just rolling the damn thing in MPI yourself. I'm very excited to see where distributed tf will be in a year or two, but it's a chore today.

Depending on how much these TPUs and other Cloud ML Engine developments help, I'd gladly abandon the attempt to roll it myself with the distributed tf.

The hope is that using Google's secret sauce to auto-distribute the execution graphs and associated data ingestion makes things "just work". At the moment, the documentation and examples for that are a bit all over the place and require writing models to conform to the newish tf.contrib.learn.Experiment API, which is also a bit underdocumented and underexampled. Using it for very large datasets (say >tens of TB) seems to be pretty challenging at this moment (to me at least). For a lot of use cases, BigTable seems to be the ideal ingestion engine for Cloud ML tf jobs, but there's no C native API. You can use BigTable, but you can only dump complete tables into tensorflow rather than querying for relevant data (since the queries cost money, a 5000-core jobs with just a few queries per core would cost you a fortune, so the ability to query BigTable in the tensorflow reader is disabled.

At any rate, I've been banging around on it for a few weeks and am really hopeful. I will follow Cloud ML Engine's career with considerable interest.