Comment by aab0 | Hacker News Reader

Comment by aab0 | Hacker News Reader

This is great news! One of the most intimidating things about getting started with deep learning if you want to understand and extend cutting-edge work is the Tower of Babel situation: aside from mastering some quite difficult and opaque concepts, you need to learn multiple frameworks in multiple languages, some of which are quite uncommon. (Want to use Torch? You have to learn Lua. Want to use Theano or Caffe? Need to learn Python too. Need to implement optimizations? Hope you've mastered C++.)

And DeepMind's research output was a major reason to need to use Torch, and hence have to learn Lua.

But by switching over to TensorFlow, this means you now have one language to learn which is supported well by all the major frameworks - Python - and you can benefit from several frameworks (Theano, Keras, TensorFlow). So the language barrier is reduced and you can focus on the framework and actual NN stuff. Further, this will also drive consolidation onto TensorFlow, reducing the framework mental overhead. As long as TF is up to the job, and it reportedly is, this will benefit the deep learning community considerably.

I'd been wondering myself what language and framework I should focus on when I start studying NNs, and this settles it for me: Python and TensorFlow.

You're overstating things a bit. There has never been a "need to learn multiple frameworks in multiple languages." As a beginner, you pick one and go with that (as evidenced by the fact that... you've picked one!). This announcement doesn't change that situation.

Nobody needs to use multiple frameworks unless you're someone (not a beginner) who wants to be able to take the code from research papers or something.

Researchers need multiple frameworks because the feature sets aren't the same in each.

I haven't done anything with deep learning, but I worked in a research lab with others who did. On the image processing side, we prototyped code in OpenCV, both with Python and C++, and MATLAB (with toolboxes) regularly because of this.

At the end of the day they have a limited amount of time and just want to test their idea the fastest way they can.

I don't think this is the reason. You can do pretty much anything in Torch/Theano. But, researchers are lazy and writing code is only a means to an end. They will shamelessly copy/hack as much open source code as possible, so if a framework has some quick module for a desired algorithm, or if a researcher wrote their paper in some framework, and you want to build off that paper, then you'll just copy that.

That's definitely correct.

One related unsolved problem in that type of research is getting people to share their actual code. Especially when multiple universities are at the bleeding edge in a field, they often publish just enough [1] to prove their point without giving everyone else the same foundation to build on easily. Even in science it gets political... who knew.

1: i.e., just their algorithms, or their code without very useful implementation-level optimizations

In ML, you can generally email the authors and very often they will be willing to send you (their really crappy) code. Although it probably helped that I sent these emails from my academic email address.

I guess that by the time the paper is in the public domain things have moved on - it's six months from submission to publication.

> There has never been a "need to learn multiple frameworks in multiple languages."

> you can generally email the authors and very often they will be willing to send you (their really crappy) code.

Code obtained from multiple authors, or even from the same author but different time periods, is code written using multiple frameworks in multiple languages. Standardizing on Python / TensorFlow reduces the risk of cognitive load along one's journey and is likely to speed up the field. If speed is what the field was missing :)

> Nobody needs to use multiple frameworks unless you're a researcher or something who wants to be able to take use the code from other people's papers or something.

I never said I wasn't. I'm interested in more than one thing, and so it makes me sad to see that the cutting-edge stuff I'm interested in is implemented in a variety of languages and frameworks (typically for no better reason than that particular researcher uses that particular framework), which means I either need to invest a huge amount of time and effort into becoming a polyglot or abandon any more in depth understanding than the paper.

I agree with you; your original comment was talking about beginners specifically, though.

That being said, it's the natural state of any field moving fast. Web development has even worse fragmentation. Mobile development has about the same amount of fragmentation. There are a dozen distributed computing frameworks (Hadoop? A dozen things on top of Hadoop? Spark? Mesos? Kubernetes?).

I agree. The frameworks are similar enough that you don't really get much from switching between them. There are differences, sure, but they are far smaller than the similarities.

Can you speak for the implementing of algorithms in TensorFlow? My impression is that it's still quite difficult to introduce something interesting that doesn't already exist. This is basically the case for all the frameworks, but I've found that even though Torch is written Lua, there is much much less abstraction.

I can't speak for TensorFlow, but I've personally found Theano to be the most easily extensible among the other widespread deep/reinforcement/gradient learning libraries (Caffe, Torch, Keras, Chainer). There does seem to be a performance hit between Theano and Caffe/Torch, but it's great for prototyping.

Implementing a new operation in TensorFlow does require a fair amount of overhead, and you'll need to know CUDA if you want to create something that works on (NVIDIA) GPUs.

The "Creating an Op" how-to[1] goes over the basic steps necessary to implement an operation, but there are a couple of things missing. Notably, proper documentation for the various functions and classes used when writing operations, as well as information on writing (and registering) a Python wrapper for additional functionality.

All this said, if you want to commit your work to the GitHub master, the Google team working on the repository does an excellent job of walking contributors through the steps necessary to get their work over the hump.

[1]: https://www.tensorflow.org/versions/master/how_tos/adding_an...

TensorFlow makes some things difficult in truly baffling ways. For example, the last time I checked there's no built-in way to get gradients w/r/t say a single weight, rather you get a gradient sum. So if you want to do anything with the Jacobian (e.g. contractive autoencoders) you are better off using Theano.

Reply on news.ycombinator.com