Comment by agibsonccc | Hacker News Reader

Comment by agibsonccc | Hacker News Reader

Parent Full thread

Hi, I'm an adjunct instructor at the data science bootcamp Zipfian Academy[1].

It seems there's still a lot of confusion as to what a data scientist is.

Data Scientists are typically analysts who know some combination of matlab/python/R that come up with predictive models to achieve some sort of business objective.

This is usually related to a businesses' profit center. The work is typically anything beyond A/B testing using basic classifiers to figure out things like churn prediction, handling data quality, all the way to doing object recognition.

Data Engineers typically work on the JVM/distributed systems to handle data at scale and implement models for data scientists in production.

With respect to deep learning, I'm also the author of a java based distributed deep learning solution called deeplearning4j[2]

I think people throw around deep learning because it's the new thing, but it really can be more accurate.

This is due to not needing to do feature engineering[3] and feature extraction[4]

Also, if you're interested, I will be giving talks at both hadoop summit[5] and OSCon[6] this year around distributed deep learning if you have any specific questions as to what all of this stuff is. Data Science is an amazing field to be in right now.

For those of you who are interested in neural networks in general, Ersatz holds a great meetup (videos/tech talks recorded!)

Happy to answer questions!

[1]: http://www.zipfianacademy.com/

[2] http://deeplearning4j.org/

[3]: http://www.cs.princeton.edu/courses/archive/spring10/cos424/...

[4]: http://en.wikipedia.org/wiki/Feature_extraction

[5]: http://hadoopsummit.org/san-jose/speakers/

[6]: http://www.oscon.com/oscon2014/public/schedule/detail/33709

[7]: http://www.meetup.com/SF-Neural-Network-Afficianados-Discuss...

How can you possibly teach data science in 12 weeks?

Little late on the reply. It's non stop 9-6pm everyday for 3 months. This would be you quitting your job to learn the whole data science stack (SQL all the way through machine learning)

Since the focus is on practical/hands on, with only the needed theory, concepts are retained enough on a practical level to become a productive junior data scientist.

Note that our acceptance rate is also very low though. We are bringing in people who already will have phds or some sort of a software engineering background. The job placement is worth it though. We are seeing avg salaries of 115-120k starting. The problems being solved are also really interesting.

Seems like a somewhat artificial distinction. I'm responsible for coming up with my own models and implementing them in production.

w.r.t. deeplearning4j: Isn't it impractical to use multiple compute nodes for parallelization, rather than GPUs when it comes to training deep nets?

Distributed GPUs ;). I'll be adding GPU support here shortly. First I'm working on the core nets, since everything is vectorized, swapping out the matrix implementations will be easy. Mainly focusing on a point release first. Just need to add Recursive Neural Tensor Nets[1]. Afterwards, distributed GPUs will be cake with the support that both NVIDIA[2] and AMD[3] support.You will find distributed compute is what google does[4]

Edit: I'd like to mention ersatz[5] here as well.

Edit: Thanks for the correction David.

[1]: http://nlp.stanford.edu/sentiment/

[2]: http://www.jcuda.org/jcuda/jcublas/JCublas.html

[3]: http://developer.amd.com/tools-and-sdks/heterogeneous-comput...

[4]: http://research.google.com/archive/large_deep_networks_nips2...

[5]: http://ersatzlabs.com

Deleted comment

Sure! So my stack currently uses iterative reduce[1]. The algorithm can then be swapped out which can either be gradient descent or conjugate gradient. The default is conjugate gradient. I also have all of the knobs like adagrad, drop out,....

In general, the idea is that you split up the training in to mini batches and then average the parameters after all of them have been computed.

I will be working on downpour SGD as well.

In my case, I use akka underneath for all of the distributed computing. Everything is inherently parallelizable using an optimal batch size algorithm (very similar to hadoops). From there, it's usually just a case of figuring out what you want your matrix computations made of.

I also built it from the ground up to speak pretty much any data format.

The idea is using a DataSetIterator that speaks hdfs, local file systems,.... This allows you to handle data locality independent of the algorithms. If you want more on the architecture, here's a pretty broad overview[2]

Admittedly, not optimal, this is a very early release of the lib. After RNTN have been impled, I will be focusing on optimizations and improvements.

[1]: https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-Re...

[2]: http://deeplearning4j.org/architecture.html

Since Ersatz got mentioned, I'd like to point out that we don't do true distributed GPU yet (it's planned). Most of the models support a single GPU with the RNN models supporting multiple GPUs on the same machine. Of course, that's still way way faster than your laptop.

Distributing computation across GPUs has been addressed here: http://techblog.netflix.com/2014/02/distributed-neural-netwo... We looked at a similar scheme, but didn't see it as practical on something like AWS

Also here: http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgC... In this case, they are using infiniband, a high speed ethernet card/bus/whatever-you-call-it, in order to share data very quickly between multiple systems. This is basically like google's experiment but w/ GPUs (and thus way cheaper?)

I'll let Adam speak to his strategy. It's an open research question but that doesn't mean there aren't any options or that those options are even terribly difficult given the right resources. Of course, real world performance is another story... whether you need and/or can use all that capacity, etc.

A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1]

By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take advantage of layer properties to optimize the sharing between GPUs. Whether that reduction in data movement is enough to make it feasible for AWS/local LAN links is up for debate.

[1] http://arxiv.org/abs/1404.5997

Reply on news.ycombinator.com