akyu

>composable, hierarchical generative models and probabilistic inference. Deep neural networks are basically just huge continuous circuit approximators.

These two things are not mutually exclusive.

I'm not sure which I find more annoying; the AI hypers, or the who people who insist that machine learning is "basically just matrix multiplication". As a researcher none of it makes a difference to me either way.

No, it will still happen. The learned features will just be "overfit" to that small dataset.

This was true maybe 10 years ago. Not so much anymore.

You can use gradient descent one example at a time. It still works just fine. The gradients are more unstable, but you will still converge eventually.

Yes SVMs are still great models. The advantage neural nets have over them are that they can do automatic feature extraction. By the time you get to the last layer of a neural net, you are basically just doing a simple logistic classification, but the features coming in have been learned from all of the previous layers.

I've even seen people use pretrained ImageNet classifiers, chop off the last layer and use an SVM as the actual classifier, and it works very well for some problems.

I really love this effort. Research papers are low bandwidth way to get information into our brains. They take a lot of effort to read, even if the ideas are not particularly complicated. Often when reading complicated material, I have to come up with metaphors in my mind to make sense of it. This is somewhat of a wasted effort, as the author who wrote the material surely had metaphors for their own mind when writing, but too often they don't share these metaphors, and stick to purely technical writing. I think this is one reason why ideas like general relativity are so popular, even though the material is actually quite complicated. The average educated person can give a reasonable explanation of general relativity because the metaphors used to explain it are so powerful, even though its very unlikely they understand any of the math involved.

Well, my ego has been sufficiently deflated enough for one night....

No you're human.

It would be worth it just to have LINQ

You are right that the learning process which ANNs use is pretty much nothing like the brain. But an argument can be made that the feature representations which they learn are quite brain like. For example in visual cortex, you will find feature detectors for things like edges and whatnot, with more abstract features as you climb to higher layers. You see very similar things in Convolutional nets.

Spiking neural network models are actually very brain like, but no one really knows how to use them effectively.

I think its also important to realize that we don't have simulate the brain to do what the brain does. If you want to build a flying machine, you could model a bird with flapping wings, but its much easier to just build a fixed wing aircraft. Back propagation may be a similar case. Evolution found a very good way to learn with neurons but it may not be optimal for the things we care about. We may be better off sticking to back propagation because it works so well. Also people are investigating backpropagation alternatives which start to look more and more brain like.

Basically my point is there seems to be intrinsic characteristics of neural computation, which ANNs and real brains both demonstrate.

Maxwell wrote that Faraday was "a mathematician of a very high order" despite his lack of formal education. I always found that fascinating.

Has anyone else noticed that they instinctively build these mind palaces with file systems on your computer? I regularly find my self recalling a fact about our code base by mentally diving through the file structure.

It seems like our brains handle structure very easily.

I love spicy food so this is great news. But lately I've been wondering about the potential effect of spicy food on gut bacteria. Anyone who's had a painful bowl movement after eating some high Scoville level hot sauce knows what I'm talking about.

This seems to agree quite nicely with Jeff Hawkins theories. Mainly that our brains are primarily doing temporal pattern recognition and language is no exception according to Hawkins. By minimizing the temporal gap between related items, you minimize the size of the patterns needed to convey an idea.

It seems like theres a kind of taboo against HTM in the ML community. I guess it stems from their lack of impressive benchmarks, but I think that's kind of missing the point when it comes to HTM. Maybe HTM isn't the right solution, but I think there is a lot to be learned by using neocortex inspired models, and HTM is at least a solid step in that direction. And the work Numenta has contributed on Sparse Codings shouldn't be overlooked.

So he is basically saying Jeff Hawkins was right all along. At least on SDRs

This seems like trying to understand how a car engine works by listening to the pitch of the sound it makes. Sure you can deduce somethings, but you aren't really going to understand the engine.

Very Cool.

I did a similar project using a Neural Network instead of a look up table, similar to how deepmind did their Atari game system. The neural network adds some overhead to the training time because of backprop, so for small state spaces which the table can handle easily it is a bit overkill. But the neural network can scale up to much more complex state spaces that would be impractical to use a table for.

Learn the hotkeys/keybindings, and the equivalent in real life applications.

In other words, invest some time in learning the most efficient manner to do seemingly mundane things. It pays off in the long run.

>fleet of Teslas

These guys are on top of their buzz game

A lot of these sound like ideas that Jeff Hawkins has been pushing with his Cortical Learning Algorithm

Can you elaborate? I don't understand.