apl

Neural architecture search (NAS) is a thing! But it's almost exclusively based on meta-gradients. Again, wouldn't put my money on GAs ever outperforming gradient-based methods again.

An AlexNet/ResNet-type moment may be in the cards for GAs, but I wouldn't put any money on it. They're typically only one better than brute-force. This can be good enough (and is certainly easy to implement), but if you can get a gradient for your problem -- you should use that. And nowadays, you can typically get a gradient!

Most recent advances in the fields you mentioned were driven by gradient-based optimization (e.g., drug design, routing, or chip design: https://www.nature.com/articles/s41586-021-03544-w).

Nature can't SGD through genomes but has a metric ton of time, so evolution might be near-ideal for sexual reproduction. We typically don't have billions of generations, trillions of instantiations, and complex environments to play with when optimizing functions... It's telling that the fastest-evolving biological system (our brain!) certainly doesn't employ large-scale GA; if anything, it probably approximates gradients via funky distributed rules.

EDIT: The most modern application I can think of was some stuff from OpenAI (https://openai.com/blog/evolution-strategies/). But the point here is one of computational feasibility -- if they could backprop through the same workload, they would.

Several hints here are severely outdated.

For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages.

Finally, buying a local cluster of TITAN Xs is an outright weird recommendation for massive models. VRAM limitations alone make this a losing proposition.

Hard disagree. V100s are a perfectly valid comparison point. They're usually what's available at scale (on AWS, in private clusters, etc.) because nobody's rolled out enough A100s at this point. If you look at any paper from OpenAI et al. (basically: not Google), you'll see performance numbers for large V100 clusters.

You can almost 1:1 translate this by swapping "tf" and "torch". No need to use nn.Conv2d -- there's a functional API for all these layers:

https://pytorch.org/docs/master/nn.functional.html#conv2d

Torch doesn't have "same" padding, so you have to manually calculate the correct padding value for your input/output shapes.

> gradient descent no longer has to be written by hand

Nobody's been writing derivatives by hand for 5+ years. All major frameworks (PyTorch, Tensorflow, MXNet, autodiff, Chainer, Theano, etc.) have decent to great automatic differentiation.

The differences and improvements are more subtle (easy parallelization/vectorization, higher-order gradients, good XLA support).

Mainly because it is genuinely exhausting for any medical practitioner. That lots of patients "enjoy" googling symptoms and coming up with far-fetched self-diagnoses is a given. But couple that with the perceived intellectual superiority of (software) engineers and you get a recipe for disaster. It's the equivalent of a doctor leaning over your shoulder while you're coding and telling you to remove random keywords.

For this particular problem, Mask R-CNN would have been the way to go -- it spits out instances as opposed to just deciding, for each pixel, to which class it belongs. Or an SSD (if we don't care about the mask at all).

There's many perspectives on everything. Deep ConvNets, for instance, can be expressed as a continuously evolving ODE. Here's a fantastic paper on this view:

https://papers.nips.cc/paper/7892-neural-ordinary-differenti...

Yeah, that's right. Doing too little proofreading with HN comments...

Their applications are tabular data (for which MLPs have never been the method of choice) and MNIST (which I could classify at 85% with a rusty nail), so it's not super impressive.

NNs and the associated toolkit shine with structured high-dimensional data where CNNs, RNNs, or modern shenanigans like Transformer networks excel. I sincerely doubt that these networks turn out to be reducible to polynomial regression in any practically useful sense of the notion. But who knows.

This represents the absolute worst kind of science journalism, completely devoid of context and domain knowledge. Virtually every definitive statement in here is wrong. Their explanation of spiking alone is a complete disaster.

Out of all the modalities, vision is easily the one we know the most about. And we do so at a fairly deep level. The discussed work seems fine but it's not the groundbreaking insight it's made out to be. Great PR work from the involved scientists (or their enterprising university marketing department).

Running Windows is perfectly fine; the major libraries for GPU-accelerated autodiff and networks (CUDNN with Pytorch or Tensorflow) have great support nowadays. It's the AMD GPU that remains essentially useless, as of 2019. If you want to get into the game, I'd recommend buying a middle-of-the-road NVIDIA GPU like the RTX2060.

For toying with autodiff and basic CNNs, CPU works just fine by the way...

That's not quite how the process works. These papers go through multiple (> 2) revisions. At any iteration, there's ample opportunity for updating references. This applies double given how long the Cueva/Wei work has been available (preprint & CCN'17 (?) contribution).

It's certainly an interesting paper, but there's a bit of publication weirdness at play here.

In October '17, Cueva & Wei put out a(n anonymous) paper that recapitulates the core result almost exactly -- that training a recurrent neural network to perform dead reckoning/path integration gives you intermediate units whose place fields strongly resemble grid cells. Critically, this only happens when regularization is applied; Cueva/Wei used noisy inputs and DeepMind implemented 50% stochastic dropout in the intermediate linear layer. There are some superficial differences (generic RNN units vs. LSTM), but at their core these studies are virtually identical. Check it out:

https://openreview.net/forum?id=B17JTOe0-

What I don't get -- why doesn't DeepMind acknowledge this result? Sure, the Nature paper was submitted in July '17, but these things go through many revisions. Clearly, DeepMind went a bit further with the whole integrating visual CNNs/grid cells part. Nonetheless: Fig. 1 is the core result, everything from Fig. 2 onwards is nice-to-have but not essential, and I feel like Cueva/Wei got there first.

Ah, well. At least the minor controversy brings in great publicity for the Cueva/Wei paper.

> I've noticed that when I get a blink/flinch response from > something (usually some sand or a bug hitting my face when > I'm on the bike), it feels like I blink just a split second > before the thing hit me.

We know very little about conscious perception or even the locus at which sensory signals are integrated to generate a conscious percept. But it's perfectly possible that delays differ across modalities and that the proprioceptive signal about lid-closing reaches whatever-relevant-area before your visual system catches up.

> I read something once about our perception of audio being > delayed by ~100ms so that it synchs up with our perception > of vision

Not an expert on audition, but the brain is really good at generating coherent representations of the physical world across modalities. I wouldn't be surprised if such cross-sensory synchronisation happened in some form.

As someone who works in visual neuroscience, this article's a tough read. Lots of statements that are semi-accurate at best.

1) Eyes don't work like cameras; there's no real "exposure" phase as such (even though there's lots of thresholds). So it's misleading to talk about discrete images that we sample at some fixed frequency. Instead, it's much more helpful to think of photoreceptors and subsequent processing stages as continuous band-pass filters. At some point, high frequencies are simply cut off because the electro-chemistry of the cell can't keep up. For us, that cut-off comes earlier than it does for invertebrates.

2) There's no mechanical interaction between light and photoreceptor. Instead, the transduction cascade of the dipteran eye seems to encompass a mechanical (as opposed to biochemical) step.

3) It's pure conjecture to talk about a fly's slowed down "perception" of the world. The reason why they take off before you get to them is much simpler -- there's a highly optimized reflex that connects eye and flight muscles via the giant fiber (a particularly rapid nerve). We have similar responses, like eye lid closing etc. Additionally, their photoreceptors are sensitive and fast. But there's zero evidence that flies have any sense of continuous time that could be faster than ours.

Ah, well. The perils of science journalism.

Thanks for coding this up. The code in the blog post is, at best, incredibly misleading.

I only skimmed the notebook, but the code looks fairly inadequate. The pandas portion, for instance, treats the data frame as a dumb array and critically ignores grouping functionality which should offer a tremendous speedup. Moreover, for the exact same task pandas should never be slower than NumPy.

Benchmarks are hard to get right but this one falls way short of saying anything at all about performance penalties incurred by various libraries and abstractions.

We didn't have to reverse-engineer the CPU. Also, CPUs are rationally designed; we have zero guarantees that an evolved brain follows any principles at all. Every brain region may be highly specialised to a single algorithm or task with zero mechanistic overlap. Figuring out deep neural nets is a closer analogy, and at the moment we have very little intuition for how to do even that. Keep in mind that the brain is vastly more complex than even the most advanced ANNs.

On the technical side: The most advanced neural techniques allow for parallel measurement calcium signals in ~10,000 neurones. That's a long way off from complete observation!

This came out two days ago and uses what they call intelligent synapses to improve multi-task learning: https://arxiv.org/abs/1703.04200

Seems closely related.

Try this: https://www.ncbi.nlm.nih.gov/pubmed/26234212

I guess the key lesson is -- don't rely on a single approach, because its limitations may well lead you astray. Applies to connectomics, physiology, modelling, etc.

Even in these apparently simple feedforward sensory networks, connectomics haven't been the anticipated panacea. There's been a flurry of follow-up papers to Takemura et al., essentially refuting the suggested model.

Turns out, even where they should connections don't constrain circuits to a sufficient degree.

Cortical layers are in no way equivalent to DNN layers.

That'd be interesting, but once you reach an orchestra you're generally beyond technical training. Players are expected to know how to play; individual training is rare and focuses on style aspects more than technical things.

The key bottleneck with all connectome projects is still reconstruction. There's on the order of a hundred billion neurones in the human brain; they're connected via 1,000-10,000 times the number of synapses. Current machine learning approaches are decent, but still require massive amounts of proof-reading.

Circuit neuroscience in Drosophila or the mammalian retina suggests neurones and synapses are indeed the level of detail we'll need in order to gain insight into computational mechanisms. So we're a long, long way off -- don't be too impressed by this particular dataset (which is insanely coarse!).

Examples of this type are not a good guideline for policy decisions, almost by definition. For every Marshall, there's a million cranks whose ideas are rejected by the establishment and rightfully so. At the outset, all these ideas appear equally wild. But resources are scarce -- we can't just fund everything.

So what to do with a billion Euros? Obviously, we shouldn't put everything in a single pot. Particularly not in one crazy pot. That's why people complain about Markram's ego trip and enabling politicians. If you got a certain amount of funding, diversify: fund a couple of cranks, sure, but 90% should go toward what's often called normal science where small steps add up to comparatively predictable progress.

(In Markram's/the HBP's defence: Ultimately, the project will yield a metric tonne of normal science, especially in terms of infrastructure etc. However, everybody will be disappointed at the end...)

A simple explanation for this: cutting carbs is tricky and requires a lot of careful planning etc., increasing the likelihood that people actually stay under their maintenance caloric intake. It doesn't really matter how you do it. As long as your expenditure is larger than your intake, you'll lose weight.

In any case, your point doesn't say anything about the healthiness of carbohydrates.

The Julia FFI for Python is absolutely excellent -- calling a particular Python library from Julia takes very, very little effort. As a language for scientific programming, Julia is way ahead of Python. So I'm not sure if this particular argument holds much water.

https://github.com/stevengj/PyCall.jl

People are trying to accomplish precisely that: it's called eLife. Wellcome Trust, HHMI and the Max-Planck-Society are backing it; adoption is moderately successful.

So there's hope!