Comment by ckcheng | Hacker News Reader

Comment by ckcheng | Hacker News Reader

Parent Full thread

For clarity, Universal approximation doesn't mean ANNs can approximate just absolutely any function...

Depending on which proof you're looking at, ANNs can provably approximate any continuous function, or any Lebesgue integrable function, or any real vector-valued continuous function on a compact subset of R^n space [1]. That's certainly a very large class of functions, but it's not just any function.

Also, those proofs only work if you allow ANNs with arbitrary number of layers or layer width. Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth. At what point is it beyond impractical? What if the ANN size (number of layers or neurons) is beyond the number of particles in the galaxy? Maybe it's just the wrong approach.

I haven't studied those proofs in detail, but as existence proofs, they just show that some ANN exists that can approximate a given function (of a certain class). There's no guarantee that there's a general algorithm that can learn the required ANN, is there? Maybe you just have to be really lucky to randomly initialize onto the correct approximating ANN...

Also, I'm not aware of any proof that human or bio NNs implement a Lebesgue integrable function, or etc. It's entirely possible bio NNs uses some weird (quantum?) physical process, and implements a function outside the scope of any ANN universal approximation theorem.

I'm just a little more cautious around those universal approximation theorems. I'm more or less in agreement with your comment otherwise!

[1]: https://en.wikipedia.org/wiki/Universal_approximation_theore...

> Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth

Perhaps there's a teapot orbiting Mars. It's unlikely tho. I think a better hypothesis would be that to model a brain (if we knew how) one would need the same order of magnitude of neurons and connections. This is completely unachievable now, and even if it were technically possible nobody has a faintest clue about how to train such a thing. We do not understand _many_ things about how the brain learns (no backprop!), how it retains and retrieves memories, etc, etc. The best studied system is vision, and computer vision does actually work pretty well by now, achieving superhuman results on some tasks. Thus far the most complex brain we've been able to accurately and completely model is not even a brain at all - it's just a handful of neurons of a worm. I took an online course in neuroscience a few years ago hoping to gain some insight for my ANN work. The situation with our understanding of even primitive brains of i.e. rodents is, well, primitive. We also don't really have math to accurately model more than a handful of real, biological spiking neurons - PDEs get way, way too complex for anybody to handle. But one could argue we don't have to: ANNs effectively operate in the frequency domain instead, where magnitude of activation approximates spike train frequency, and timing effects are ignored.

Don't get me wrong, I'm pretty sure if we do achieve real AGI, it likely won't be with anything even remotely like the current tech and science stack. Energy efficiency and compute density is just not sufficient. But I'm also pretty confident that even with the current stack we will be able to, eventually, approximate intelligence closely enough to suit many practical uses that seem completely out of reach today, including even some limited degree of logical inference, and dare I say, cognition.

If only people (including myself) weren't so chickenshit and actually seriously worked on (and funded) this stuff, that could even happen in my lifetime. As things stand right now, people are afraid to touch anything AGI related due mostly to fear of ridicule.

In my comment above I am merely referring to the conjecture that _if_ we had ANNs that are large enough, and _if_ we knew how to structure and train them, there doesn't seem to be anything that would _fundamentally_ prevent them from supporting AGI. Those are very, very massive "ifs" indeed, without even a faintest glimmer of light at the end of the tunnel.

I completely agree with your conclusions starting from these ifs!

However, I would add that another interesting area of study that is essentially not being explored at all is exactly to try to understand more about human cognition at the computational level (whereas neuroscience is working at the biological level). Basically, it seems we have abandoned any work in AI that would try to model human learning or human understanding of language in a human-understandable way, and instead we are aiming for reproducing human behaviors for engineering purposes.

Reply on news.ycombinator.com