aidanrocke

Whenever the discussion of liberty versus equality of outcomes arises there is a tendency to create a false dichotomy where we are forced to choose between extreme positions that are mutually exclusive. However, this is a mistake as the best approach is to find a compromise between different tradeoffs.

From the abstract:

'The brain carries out tasks that are very demanding from a computational perspective, apparently powered by a mere 20 Watts.This fact has intrigued computer scientists for many decades, and is currently drawing many of them to the quest of acquiring a computational understanding of the brain. Yet, at present there is no productive interaction of computer scientists with neuroscientists in this quest.'

For more information, check this brilliant thread by Michelle Kendall: https://twitter.com/mishkendall/status/1239946522390867969

This article uses a Julia implementation to explain how the complex-step method allows us to easily compute derivatives of analytic functions up to machine-precision.

Possibly the best short blog post on this subject on the internet.

tl;dr

1. The objective of this paper is to show how the complex-step derivative approximation is related to algorithmic differentiation.

2. For a tutorial on complex-step differentiation, I can recommend the blog post of John Lapeyre: http://www.johnlapeyre.com/posts/complex-step-differentiatio...

tl;dr

1. Within the context of optimisation, differentiable approximations of the min and max operators on R^n are very useful.

2. However, in order for these approximations to be useful they must also be numerically stable.

tl;dr

1. Here I present an elementary proof for a classical result in random matrix theory that applies to any random matrix sampled from a continuous distribution.

One of its many important consequences is that almost all linear models with square Jacobian matrices are invertible.

2. This is also relevant to scientists that want stable internal models for deep neural networks since a deep network is an exponentially large ensemble of linear models with compact support.

tl;dr

1. The typical deep neural network tutorial introduces deep networks as compositions of nonlinearities and affine transforms.

2. In fact, a deep network with relu activation simplifies to a linear combination of affine transformations with compact support. But, why would affine transformations be useful?

3. After recent discussions on Twitter it occurred to me that the reason why they work is that they are actually first-order Taylor approximations of a suitable analytic function.

4. What is really cool about this is that by this logic partial derivatives, i.e. Jacobians, are computational primitives for both inference and learning.

5. I think this also provides insight into how deep networks approximate functions. They approximate the intrinsic geometry of a relation using piece-wise linear functions.

This works because a suitable polynomial approximation exists and all polynomials are locally Lipschitz.

For the kind of explosiveness that man will be able to contrive by 1980, the globe is dangerously small, its political units dangerously unstable.-John von Neumann

I mean mimetic behaviour in the sense of René Girard.

So if I had to write a tl;dr I would say that there are two economic incentives for neuromorphic computing:

1. Energy efficiency: More brain-like computers will eventually an energy efficiency comparable to the human brain. Consider that AlphaGo Zero used ~200 KiloWatts for learning Go vs. a human that uses about ~20 Watts for learning everything that humans can learn.

2. AI & Neuroscience research: What better way to test neuroscience theory than to physically build a brain? As the saying goes, if you can't build you don't understand it.

A vertex prediction model implementation based on https://arxiv.org/abs/1704.05548

A collection of pre-trained RL agents using Stable Baselines http://stable-baselines.readthedocs.io/ developed by Antonin Raffin(https://araffin.github.io/) and Ashley Hill(https://github.com/hill-a).

For context, this occurred to me when I was trying to find a way to apply policy gradients to tic-tac-toe. I haven't compared it to the Gumbel-Softmax trick yet but empirically I can say that it works.

I'm generally interested in benchmarks and/or good tutorials on different methods for accumulating gradients in TensorFlow.

The following analysis grew out of a desire to address Elon Musk's ontological argument and the underlying Singularity hypothesis that is popular among AI researchers, AI engineers and the general public. In this essay I demonstrate that although the Singularity hypothesis has the form of a scientific theory, with its share of graphs(always exponential curves), confabulations(i.e. cherry-picked data), and shoddy analysis; in substance the Singularity is nothing more than a religion. Moreover, it is the first scientific theory which rests upon a single unobserved data point which is both imminent and perpetually beyond the grasp of scientists. In spite of this, the theory has recently gained traction due to the combined influence(and aura of credibility) of Elon Musk, Nick Bostrom, and Stuart Russell who have managed to conflate real AI risks with imaginary ones emanating from the Singularity narrative.

I appreciate your feedback.

The paper I referenced by Josef Urban et al. is representative of the state-of-the-art. That said, this is a first draft and I will make a lot of changes. Adding more references on ATP being one of them.

If you have any other thoughts/comments, I'd be very interested in hearing them.

That's a very good point. I also think there's a difference between Automated Theorem Proving and AGI since entities with high-functioning general intelligence(i.e. AGI) aren't necessarily good at mathematics. ATP is a specific form of AGI in my opinion.

Good morning Smerity,

I'm sorry if my response sounded harsh. It wasn't my goal to confuse readers and I admit that it's a work in progress. That said I must clarify a few things as I realise that I left out important details:

1. Regarding neuroscience: Those researchers I've mentioned(Kording et al.) are actually people I've discussed this issue with recently. That's the context. However, I must say that I thoroughly agree with you that "deep learning learns just like a human brain" is a very inaccurate statement and a reflection of the imprecise understanding many deep learning practitioners(and some researchers) have of their own field.

2. My approach to math: Was certainly not the approach used by deep learning tutorials. The issue I have with the proliferation of these tutorials on the internet is that they lead to familiarity rather than understanding. How to reach such a theoretical understanding is debatable and I'd like to encourage a debate.

In fact, this list is part of a bigger project which I'd like to collaborate on with others in the near future. You can think of it as the early stages of a Bourbaki project for deep learning. :)

Who was Bourbaki: http://www.ams.org/notices/200709/tx070901150p.pdf

Hello Smerity,

I have read the deeplearning book and I think it's great that you know Yarin. His PhD thesis is the closest thing to a survey paper on uncertainty in Deep Learning. I also think that while the original dropout paper is nice, Yarin's paper extends this method in a very useful manner.

Now, let me address your other points:

1. This is not for people who want to get started in deep learning. Or it could be. Do you know how I got started into mathematics?

2. The dubious neuroscience connection isn't dubious at all. In particular, I would also advise that you read 'Towards an integration of deep learning and neuroscience' by Marblestone et al. If you still disagree, I could assure you that Dr Kriegeskorte, Konrad Kording, Professor Bengio and many others are of a different opinion.

Furthermore, the language of statistical learning has shown its limitations. Consider Srivastava's paper on Locally Competitive Networks and every other paper on deep rectifier networks for example. As the complexity of deep models approach those of biological systems, I have no doubt we'll be using similar tools to understand these models.

fyi: I'm currently working on deep rectifier networks.

Aidan

tl;dr:

The main point I’m trying to drive home is semantic in nature. There is no hard and fast rule that AI has to be digital or that it must be programmed via a laptop. The key thing is that there are universal design principles for building substrate-independent AI systems.

A lot of people in the comments below are fussing over whether there is a Muslim ban. Well, think about it for two seconds.

1. This doesn't set a good precedent. 2. Donald Trump is aiming for a Muslim ban. He said so during the elections.

He said it during the elections: http://www.independent.co.uk/news/world/americas/donald-trum...

At this point I'm surprised by how many people are bickering over this. Why don't you guys just say that you voted for Donald Trump? That would simplify the discussion.