Hacker News Reader

Comment by p1esk | Hacker News Reader

Nonetheless, I get excited when I come across research like this

I don't, even though I did a phd in this type of research (analog neural computation with novel memory technologies). You either design specialized HW for the best available algorithm to solve a task of interest, or design better algorithms for that task. Do not try to design better hardware for poorly understood algorithms, especially when these algorithms don't really work (82% on MNIST, really?). If the task of interest is image classification, either focus on accelerating convnets, in which case all you really need is a fast multiplication of large matrices with enough precision, or focus on developing better algorithms (capsules, attention, neural trees, etc), in which case you use GPUs until you beat the state of the art.

View on HN

If your goal is to build something that works and is of interest to industry ($$$), I couldn't agree more :-)

But I don't think that's the goal here. I view this type of research as part of early efforts in a long-term quest to discover if it's possible to build "neuromorphic" devices capable of universal computation that are much more efficient than current methods. These efforts today seem to be roughly at the same stage that "deep learning" was in the late 1970's to early 1980's, or even at an earlier stage.

For what it's worth, I also think it's worthwhile to develop hardware/software that accelerates linear/multilinear algebra operations and/or makes it practical for us to use more general/flexible abstractions in our code.[a]

[a] For example, I would love to be able to model each artificial neuron as a separate object in code, making it trivial to group/ungroup neurons into irregularly shaped and/or overlapping groups and easily shrink/expand/alter such groups on the fly without incurring a performance penalty. Right now, that's not possible: If we want decent performance, we have little choice but to pigeonhole all our designs into functions that operate on grid-like shapes (equal-sized vectors, matrices, higher-order tensors) and resort to thinks like masking so we can have fast multiply-sum operations.

What is "universal computation"? Turing complete? Then all we need is a good NAND gate. Or whatever computation is happening inside our brains? Then maybe we should first figure out what is actually happening there.

The field today seems to be roughly at the same stage that "deep learning" was in the late 1970

Have all the research directions that seemed promising in 70s become as successful as DL?

My point is - if you're interested in how brains work, focus on that. If you're interested in building better hardware, focus on that instead and try to build something that can be useful. This article got published because of "neuromorphic" (such a bullshit term!), and because there are MNIST results (which are embarassing, but the peer reviewers didn't catch that because they are probably device physicists, not ML people)

EDIT (responding to your edit): I would love to be able to model each artificial neuron as a separate object in code

So you want to run some experiments, without any idea whether it would lead to any good insights or benefits. Seems like building a specialized hardware just to accelerate this particular experiment is premature, to say the least. Just rent a bigger GPU cluster - you'll get your answer much faster and much cheaper.

Similar arguments could have been made against deep (multilayer) neural networks (DNNs) for a couple of decades.

No one knew for certain in the 70's and 80's if DNNs would pay off, yet a handful of then-obscure researchers persisted. And for quite a while DNNs were kind of a research backwater -- some researchers moved to Canada because they couldn't get funding anywhere else. As a close approximation, "almost no one" was interested in DNNs until Alexnet won ImageNet in 2012.

Just because we don't know which research directions may or may not pay off decades from now does not mean such directions aren't worthwhile and should be abandoned.

Reply on news.ycombinator.com