Hacker News Reader

Comment by moglito | Hacker News Reader

"considering that 2016 is generally regarded as the date of the deep learning revolution" --

I thought it was 2012, when AlexNet took the imagenet crown?

That's probably fair. But you'd be hard-pressed to find a DL stack to try out your ideas with prior to 2016, since that's when Tensorflow launched. :)

(Gosh, it's been less than a decade. Time sometimes doesn't fly, considering how much it's changed the world since then...)

Theano was first released in 2007.

That’s actually fascinating. Were there many experiments done in it back in the 00’s?

I’m just trying to imagine the things you could do with it back then. 2007 had relatively fast gpus for the time, but certainly nothing compared to today. Yet it’d certainly be enough for MNIST training, which makes me wonder what else could be done.

FWIW in 2016 I was at an ML team at Apple that had been shipping production neural networks on-device for a while already. At the everyone used an assortment of random tools (Theano, Torch, Caffe). I worked on an internal tool that originally started as a Theano fork but was closer to a modern-day Tensorflow XLA (and has since been axed in favor of Tensorflow for most teams).

You can look at Yoshua Bengio's Google Scholar profile [1] and scroll down to see what they were working on around that time.

Here are some papers with many citations:

- An empirical evaluation of deep architectures on problems with many factors of variation [2]

- Extracting and composing robust features with denoising autoencoders [3]

- Scaling learning algorithms towards AI [4]

[1] https://scholar.google.com/citations?hl=en&user=kukA0LcAAAAJ...

[2] https://scholar.google.com/citations?view_op=view_citation&h...

[3] https://scholar.google.com/citations?view_op=view_citation&h...

[4] https://scholar.google.com/citations?view_op=view_citation&h...

Yep, I worked on a production DL system based on Theano in ~2014

Reply on news.ycombinator.com